Artificial intelligence processing device and method and related equipment

Abstract

The embodiment of the invention discloses an artificial intelligence processing device and method and related equipment. The device comprises a data interface and a processing unit. The data interface is used for receiving a first matrix, and the first matrix is a feature map matrix; the processing unit is used for splitting each matrix element in the first matrix into a plurality of matrix elements based on a plurality of first bit width values so as to obtain a plurality of first sub-matrixes, and the sizes of the plurality of first sub-matrixes are the same as the size of the first matrix; each first sub-matrix in the plurality of first sub-matrixes and each second matrix in at least one second matrix are calculated to obtain a plurality of sub-results, and the at least one second matrix is a weight matrix; and splicing the plurality of sub-results to obtain an output result. According to the artificial intelligence processing device, the quantitative configuration strategy supporting flexibility can be newly added on the basis of fixed quantitative configuration provided by hardware under the condition that the hardware is not changed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an artificial intelligence processing device, method and related equipment. Background Technology

[0002] Large Language Models (LLMs) have made groundbreaking progress in recent years, performing exceptionally well on various tasks and possessing outstanding task processing capabilities, making them a key area of ​​focus for both academia and industry. However, LLMs face enormous challenges stemming from their massive computational and storage requirements. For example, the Generative Pre-Trained (GPT) 175B version has a staggering 175 billion parameters, requiring at least 320GB (using multiples of 1024) of half-precision (FP16) format storage. Deploying this model for inference also requires at least five A100 Graphics Processing Units (GPUs), each with 80GB of memory, to ensure efficient operation.

[0003] In existing technologies, model quantization is a common technique for achieving model compression and inference acceleration. Extremely low bit quantization converts the weights of the LLM model from floating-point numbers to low-precision fixed-point numbers, which can transform large, resource-intensive LLMs into compact versions suitable for storage on limited mobile devices, significantly reducing storage requirements and computational complexity, and greatly lowering the threshold for deploying LLMs on the edge.

[0004] Due to the decoder-only nature of LLM models, LLM computation during incremental inference is typically memory-intensive, unlike the computationally intensive nature of traditional Convolutional Neural Networks (CNNs). Furthermore, LLM memory access is primarily concentrated on weights, not feature maps. Therefore, LLM quantization typically employs asymmetric quantization, where feature maps use higher quantization bit widths (e.g., 16-bit or 8-bit) to maintain precision, while weights use lower quantization bit widths (e.g., 4-bit or 3-bit). However, existing neural network acceleration units are primarily designed for CNNs and typically only support 8-bit or 4-bit feature map / weight computations, not the 16-4 quantization configurations required by LLM. When hardware does not support this quantization strategy, weight quantization is generally rolled back to a higher bit count, leading to performance degradation and increased power consumption during LLM inference. Summary of the Invention

[0005] This application provides an artificial intelligence processing device, method, and related equipment, which can add support for flexible quantization configuration strategies on the basis of the fixed quantization configuration provided by the hardware without changing the hardware.

[0006] In a first aspect, embodiments of this application provide an artificial intelligence processing apparatus, which may include a first data interface and a processing unit; the first data interface is used to receive a first matrix, wherein the first matrix is ​​a feature map matrix; the processing unit is used to: split each matrix element in the first matrix into multiple matrix elements based on multiple first-width values ​​to obtain multiple first sub-matrices, wherein the size of each of the multiple first sub-matrices is the same as the size of the first matrix; calculate each of the multiple first sub-matrices with each of at least one second matrix to obtain multiple sub-results, wherein the at least one second matrix is ​​a weight matrix; and concatenate the multiple sub-results to obtain an output result.

[0007] This application provides an artificial intelligence (AI) processing device applied in AI inference scenarios. By performing online splitting of matrix elements in the feature map matrix during matrix operations, the model quantization configuration is adapted to the hardware bit quantization configuration of the AI ​​processing device. This achieves software support for non-standard quantization configurations on existing edge AI hardware specifications. Specifically, this application design provides an AI processing device that can split matrix elements online based on configuration parameters (i.e., multiple first-bit width values), thereby enabling arbitrary combinations of quantization bit numbers for the feature map matrix. This solves the technical problem that existing AI hardware typically only provides fixed quantization configurations for users. For example, some AI hardware supports INT8-8 and INT4-4 quantization, but the feature map and weights to be calculated are INT16 and INT8 respectively. With the splitting method described in this application, the INT16 feature map matrix can be split into two INT8 matrices, adapting to the hardware's bit quantization configuration and enabling flexible combinations of feature map and weight quantization bit numbers. In summary, this application provides an artificial intelligence processing device to support a flexible quantization computing scheme with flexible quantization configuration. When the model is deployed on the edge, a quantization configuration friendly to the AI ​​model (such as an LLM model) can be reconstructed through the quantization configuration supported by the hardware. That is, without changing the hardware, based on the fixed quantization configuration provided by the hardware, additional quantization strategies can be added through software. In some application scenarios, this can reduce bandwidth and power consumption, and improve computing speed. For example, when applied to LLM scenarios, it can reduce the usage of LLM's read-only memory (ROM) and random access memory (RAM), thereby improving LLM inference performance.

[0008] In one possible implementation, the plurality of first bit width values ​​are A1, A2...Ax, where x is an integer greater than 1. The processing unit is specifically used to: split each matrix element in the first matrix into x matrix elements with bit widths of A1, A2...Ax, to obtain x first sub-matrices with the same size as the first matrix but different bit widths. In this embodiment, the processing unit can specifically split each matrix element in the feature map matrix, specifically, splitting each matrix element into x matrix elements with bit widths of A1, A2...Ax, to generate x first sub-matrices with the same size as the first matrix but different bit widths. For example, if the first matrix is ​​an M x K matrix, the first sub-matrice is defined as follows: For the matrix element A in the first row and first column of the first matrix... 11 Arrange A in order from left to right. 11It is split into matrix elements with bit widths of A1, A2...Ax, respectively A1 11 A2 11 ......Ax 11 For matrix element A in row 2 and column 1 21 Arrange A in order from left to right. 21 It is split into matrix elements with bit widths of A1, A2...Ax, respectively A1 21 A2 21 ......Ax 21 And so on, for the matrix element A in the Mth row and Kth column... MK Arrange A in order from left to right. MK It is split into matrix elements with bit widths of A1, A2...Ax, respectively A1 MK A2 MK ......Ax MK Finally, according to the original positions of the matrix elements in the first matrix, the M rows * K columns of element A1 with a bit width of A1 are... 11 A1 21 A1 31 ......A1 Mk As a first submatrix, the M rows * K columns of element A2 are... 11 A2 21 A2 31 ......A1 Mk As a first submatrix, and so on, the M rows * K columns of element A1 of width Ax are used. MK A2 MK ......Ax MK As a first submatrix, x first submatrices are ultimately obtained.

[0009] In one possible implementation, the at least one second matrix comprises y second matrices obtained by splitting each matrix element of the third matrix into y matrix elements, wherein each of the y second matrices has the same size as the third matrix, and y is an integer greater than 1. In this embodiment, the at least one second matrix (i.e., the y second matrices) is also derived from the original weight matrix (i.e., the third matrix) by splitting its matrix elements, similar to the splitting method used for the first matrix. For example, the y second matrices can be pre-split offline or online in the AI ​​processing device. That is, in the matrix operation process of this embodiment, not only can the bit width of the matrix elements in the feature map matrix be split, but further, the bit width of the matrix elements in the weight matrix can also be split. In this way, a more flexible combination of quantization bits for the feature map and weight can be achieved.

[0010] In one possible implementation, the processing unit is further configured to: based on multiple second bit-width values, split each matrix element in the third matrix into y matrix elements with bit widths of B1, B2, ..., By, to obtain y second matrices with the same size as the third matrix but different bit widths. In this embodiment, the processing unit in the AI ​​processing device can further perform online splitting of the third matrix elements, thereby splitting it into y second matrices. That is, it splits the bit width of the matrix elements from two dimensions: the feature map matrix and the weight matrix, thus supporting more model quantization strategies through software without changing the AI ​​acceleration hardware.

[0011] In one possible implementation, the processing unit is further configured to: obtain the split quantization coefficients, the split quantization coefficients including multiple sub-quantization coefficients; and perform quantization operations on each of the multiple sub-results and one of the multiple sub-quantization coefficients to obtain the multiple sub-results after the quantization operation. In this embodiment, the quantization coefficients involved in the matrix operation can also be adaptively split according to the rules for splitting the feature map matrix or weight matrix to adapt to the quantization configuration supported by the AI ​​processing device. Specifically, each of the multiple sub-results obtained after the operation of the feature map matrix and the weight matrix is ​​processed with a quantization coefficient, that is, one sub-result corresponds to one quantization coefficient, and finally, the multiple quantized sub-results are calculated.

[0012] In one possible implementation, the processing unit is further configured to: perform sign addition processing on the unsigned sub-results among the plurality of sub-results after the quantization operation, respectively, to obtain the plurality of sub-results after the processing. In this embodiment, when the plurality of sub-results after the quantization operation includes one or more unsigned sub-results, and in some computational scenarios it is necessary to convert the operation type to a signed data type, sign addition processing can be performed on the one or more unsigned sub-results. That is, the AI ​​processing device in this embodiment can process the quantization coefficients of the plurality of sub-results obtained during the calculation of multiple first sub-matrices and at least one second matrix, then perform sign addition processing on the unsigned sub-result portion, and then adjust it through a bias term to ensure the correctness of the calculation result. In this way, it can support the conversion of the operation process of unsigned data type to the operation of signed data type, thereby supporting more types of operations without changing the AI ​​hardware.

[0013] In one possible implementation, the matrix elements in the first matrix are either positive integers or negative integers. The quantization strategy in this embodiment is applicable to scenarios where the matrix is ​​an integer or a negative integer; that is, regardless of whether the scenario involves positive or negative integers, the artificial intelligence processing device in this application can achieve a more flexible combination of quantization bits.

[0014] Secondly, embodiments of this application provide an artificial intelligence processing method, which may include: receiving a first matrix, the first matrix being a feature map matrix; splitting each matrix element in the first matrix into multiple matrix elements based on multiple first-order width values ​​to obtain multiple first sub-matrices, the size of each of the multiple first sub-matrices being the same as the size of the first matrix; calculating each of the multiple first sub-matrices with each of at least one second matrix to obtain multiple sub-results, the at least one second matrix being a weight matrix; and concatenating the multiple sub-results to obtain an output result.

[0015] In one possible implementation, the plurality of first-bit width values ​​are A1, A2...Ax, where x is an integer greater than 1; the step of calculating each of the plurality of first sub-matrices with each of at least one second matrix to obtain a plurality of sub-results includes: splitting each matrix element in the first matrix into x matrix elements with bit widths of A1, A2...Ax to obtain x first sub-matrices with the same size as the first matrix but different bit widths.

[0016] In one possible implementation, the at least one second matrix comprises y second matrices obtained by splitting each matrix element of the third matrix into y matrix elements, wherein each of the y second matrices has the same size as the third matrix, and y is an integer greater than 1.

[0017] In one possible implementation, the method further includes: splitting each matrix element in the third matrix into y matrix elements with bit widths of B1, B2...By based on multiple second bit width values, so as to obtain y second matrices with the same size as the third matrix but different bit widths.

[0018] In one possible implementation, the method further includes: obtaining split quantization coefficients, the split quantization coefficients including multiple sub-quantization coefficients; performing a quantization operation on each of the multiple sub-results and one of the multiple sub-quantization coefficients to obtain the multiple sub-results after the quantization operation.

[0019] In one possible implementation, the method further includes: performing sign addition processing on the unsigned sub-results among the plurality of sub-results after the quantization operation, so as to obtain the plurality of sub-results after the processing.

[0020] In one possible implementation, the matrix elements in the first matrix are either positive integers or negative integers.

[0021] Thirdly, embodiments of this application provide a computing device that may include an artificial intelligence processing device and a processor as provided in any of the implementations of the first aspect above, wherein the processor is coupled to the artificial intelligence processing device.

[0022] In this embodiment of the application, the computing device includes the AI ​​device provided by any of the implementations of the first aspect described above. Furthermore, the computing device also includes a processor coupled to the AI ​​device. The AI ​​device can be mounted on the processor as a coprocessor, and the processor can allocate tasks to it.

[0023] In one possible implementation, the processor is further configured to: based on a plurality of second bit-width values, split each matrix element in the third matrix into y matrix elements with bit widths of B1, B2...By, to obtain y second matrices with the same size as the third matrix but different bit widths.

[0024] In this embodiment of the application, the processor in the computing device can split the original weight matrix based on the acquired multiple second bit width values. For example, according to the bit widths B1, B2...By, each matrix element in the original weight matrix (i.e., the third matrix) can be split, and finally y second matrices with the same matrix size as the original weight matrix but different bit widths can be obtained. Then the processor can send the y second matrices to the AI ​​processing device in the computing device for subsequent matrix operations.

[0025] Fourthly, this application provides a semiconductor chip that may include an artificial intelligence processing device provided by any of the implementations of the first aspect described above.

[0026] Fifthly, embodiments of this application provide a system-on-a-chip (SoC) that may include an artificial intelligence processing device provided by any of the implementations of the first aspect described above.

[0027] Sixthly, this application provides a neural network processor that has the function of implementing any of the artificial intelligence processing methods described in the second aspect above. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0028] Seventhly, this application provides a chip system comprising the computing device provided in any of the implementations of the third aspect above. In one possible design, the chip system further includes a memory for storing program instructions and data necessary or related to the operation of the chip system. The chip system may be composed of chips or may include chips and other discrete devices.

[0029] Eighthly, this application provides a terminal device including an artificial intelligence processing device, which is the device provided in any of the implementations of the first aspect described above. The terminal device may further include a memory coupled to the artificial intelligence processing device, which stores necessary program instructions and data of the terminal device. The terminal device may also include a communication interface for communicating with other devices or communication networks.

[0030] Ninthly, this application provides a server including an artificial intelligence processing device, which is the device provided in any of the implementations of the first aspect described above. The server may further include a memory coupled to the artificial intelligence processing device, which stores necessary program instructions and data for the server. The server may also include a communication interface for communicating with other devices or communication networks.

[0031] In a tenth aspect, this application provides a computer-readable storage medium storing a computer program that, when executed by multiple electronic devices, implements the artificial intelligence processing method flow described in any one of the second aspects above.

[0032] In one aspect, embodiments of this application provide a computer program product including instructions that, when executed by a computing device, enable the computing device to perform the artificial intelligence processing method flow described in any of the second aspects above. Attached Figure Description

[0033] Figure 1A This is a schematic diagram of the structure of an intelligent question-answering system based on a large language model provided in an embodiment of this application.

[0034] Figure 1B This is a schematic diagram of another intelligent question-answering system based on a large language model provided in an embodiment of this application.

[0035] Figure 2A This is a schematic diagram of the structure of an artificial intelligence processing device provided in an embodiment of this application.

[0036] Figure 2BThis is a schematic diagram of the structure of a processing unit in the artificial intelligence processing device provided in the embodiments of this application.

[0037] Figure 2C This is another schematic diagram of the structure of the processing unit in the artificial intelligence processing device provided in the embodiments of this application.

[0038] Figure 2D This is a hardware structure diagram of another artificial intelligence processing device provided in the embodiments of this application.

[0039] Figure 3A This is a schematic diagram illustrating how a first matrix is ​​split into multiple first sub-matrices, as provided in an embodiment of this application.

[0040] Figure 3B This is a schematic diagram illustrating how a third matrix is ​​split into multiple second matrices, as provided in an embodiment of this application.

[0041] Figure 4A This is a schematic diagram illustrating how multiple first sub-matrices and multiple second matrices are calculated to obtain multiple sub-results, as provided in an embodiment of this application.

[0042] Figure 4B This is a schematic diagram of element decomposition for a positive number scene, provided as an embodiment of this application.

[0043] Figure 4C This is a schematic diagram of element splitting in a negative number scenario provided in an embodiment of this application.

[0044] Figure 5A This is a schematic diagram illustrating the overall process of splitting and calculating a feature map matrix, as provided in an embodiment of this application.

[0045] Figure 5B This is a schematic diagram illustrating the overall process of splitting and calculating a weight matrix, as provided in an embodiment of this application.

[0046] Figure 6 This is a flowchart illustrating an artificial intelligence processing method provided in an embodiment of this application. Detailed Implementation

[0047] The embodiments of this application will now be described with reference to the accompanying drawings. The terms "first," "second," "third," and "fourth," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The reference to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0048] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0049] First, some of the terms used in this application will be explained to facilitate understanding by those skilled in the art.

[0050] (1) Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theory.

[0051] (2) A Convolutional Neural Network (CNN) is a multi-layered neural network, with each layer consisting of multiple two-dimensional planes. Each plane is composed of multiple independent neurons, and these neurons share weights. Weight sharing reduces the number of parameters in the neural network. Currently, in CNNs, the processor typically performs convolution operations by convolving the input signal features with the weights, converting this into matrix multiplication between the signal matrix and the weight matrix. In the specific matrix multiplication operation, the signal matrix and weight matrix are processed in blocks to obtain multiple fractal signal matrices and fractal weight matrices. Then, matrix multiplication and accumulation operations are performed on these multiple fractal signal matrices and fractal weight matrices.

[0052] (3) Feature Map: The input, output, and intermediate computation results of a neural network model for inference. For example, in a Convolutional Neural Network (CNN), the input and output data of a convolutional layer are sometimes referred to as feature maps. Specifically, the input data of a convolutional layer is called the input feature map, and the output data is called the output feature map. For instance, if the input is a left matrix M×K, a right matrix K×N, and the output matrix M×N, and we consider matrix multiplication in the context of convolution, the left matrix is ​​the input feature of the convolution (W=1, H=1, batch=M, input channel=K), and the right matrix is ​​the weight of the convolution (R=1, S=1, output channel=N, input channel=K).

[0053] (4) Weights: The weight data of a neural network model. In deep learning, the weights of a model refer to the parameters in the Q-axis of the neural network. These parameters are used to adjust and learn the model's behavior so that it can effectively map input data and extract useful features. Deep learning models typically consist of many neurons and connections, and weights are the parameters that connect these neurons. Taking a fully connected layer as an example, if there is an input layer and an output layer, there is a weight between each input neuron and the output neuron. Assuming the input layer has n neurons and the output layer has m neurons, there are n × m weights, each weight corresponding to a connection. These weight matrices define the relationship between the input layer and the output layer.

[0054] (5) Large Language Model (LLM): A large language model is an artificial neural network that has been trained on a large amount of text data and can generate human-like text.

[0055] (6) The decoder-only architecture is a neural network model architecture specifically designed for natural language processing (NLP) tasks. Compared to the common encoder-decoder architecture, the decoder-only architecture only has a decoder component and no encoder.

[0056] (7) Bit, abbreviated as lowercase "b". Computers can convert 0 and 1 into signals in circuits for calculation. A bit is the smallest unit for storing and moving data in a computer, and this unit has only two values, 0 and 1.

[0057] (8) Byte, abbreviated as uppercase "B". Bytes are related to characters. English characters are usually one byte, or 1B. Chinese characters usually exceed 2 bytes due to character set limitations. Conversion: 8 bits equals 1 byte. One byte equals eight bits.

[0058] (9) Integer type (integer, INT), the most commonly used types in the program are the general integer type (abbreviated as "integer type" or "int") and the long integer type (abbreviated as "long integer"). The type name of the integer type is "int", and the type name of the long integer type is "long int", which can be abbreviated as "long". int and long are both keywords.

[0059] (10) A floating-point number is a numerical representation of a number belonging to a specific subset of rational numbers. In computers, it is used to approximate any real number. Specifically, this real number is obtained by multiplying an integer or fixed-point number (i.e., the mantissa) by an integer power of a certain base (usually 2 in computers). This representation is similar to scientific notation with a base of 10. A floating-point number a is represented by two numbers m and e: a = m × b^e.

[0060] (11) Fixed-point numbers are a method of representing numbers used in computers. The position of the decimal point in the numbers involved in the calculation is fixed. They include fixed-point integers and fixed-point decimals. Fixed-point integers are those where the decimal point is after the least significant bit. If the exponent J = 0 and the mantissa S is a pure integer, then it is called a fixed-point integer. The decimal point of a fixed-point integer is implicitly contained in different positions by prior agreement. Fixed-point decimals are those where the decimal point is fixed after the most significant bit. If the machine word length is n+1 bits and X = X0.X1X2...Xn, then the decimal represented is (-1)X0*((X1²(-1))+X2²(-2)+...+Xn-1²(-n+1)+Xn²(-n)). Fixed-point numbers can represent both integers and decimals. The range and precision of the representation are affected by the position of the decimal point. In modern computers, fixed-point numbers are usually used to represent integers. Compared to floating-point numbers, fixed-point representation has the advantage of faster processing speed, making it suitable for applications with high real-time requirements, such as digital signal processing and image processing. However, the disadvantage of fixed-point representation is its limited range and precision, making it less flexible than floating-point numbers.

[0061] (12) Int8, equal to Byte, occupies 1 byte, and its value range is from -128 to 127; Int16, equal to short, occupies 2 bytes, and its value range is from -32768 to 32767; Among them, 1. Difference in representation range: int8 is a signed integer type that uses 8 bits to represent, and int16 is also a signed integer type that uses 16 bits to represent, with a value range of -32768 to 32767; 2. Difference in space occupied: int8 occupies 1 byte, that is, 8 binary bits, and each binary position can store 0 or 1, while int16 occupies 2 bytes, that is, 16 binary bits, which has more positions that can be used to store digital information and expand its value range.

[0062] First, to facilitate understanding of the embodiments of this application, the specific technical problem to be solved by this application is further analyzed and proposed. Currently, there are various technical solutions for implementing model quantization configuration; one of them is exemplarily listed below:

[0063] Option 1: Multi-level Management Solution

[0064] In existing edge-side neural network hardware acceleration units, fixed quantization configurations are typically provided for user use. They usually only support standard 8-bit or 4-bit quantization for both feature maps and weights, or partial combinations of quantization bit counts, such as some hardware supporting INT16-8 quantization. However, it is difficult to directly support arbitrary combinations of quantization bit counts for feature maps and weights, such as INT8-4 or INT16-4 quantization.

[0065] When it is necessary to support configurations such as INT8-4 and INT16-4, the only option is to use the upfill method, such as upfilling INT8-4 calculations to INT8-8 calculations.

[0066] The disadvantages of Option 1:

[0067] Using weight upfilling has obvious drawbacks for LLM because the scale of LLM weights can reach the billion level. If such a large number of weights are processed every frame, the power consumption and performance overhead will be very large. If they are processed once and then temporarily stored in the processor's memory, the processor's memory read and write will be under great pressure, and the power consumption of memory read and write will double.

[0068] In summary, the existing quantization configuration schemes have shortcomings. In existing edge-side neural network hardware acceleration units, computation can only be performed based on fixed quantization configurations, and flexible configuration according to different computational needs is not possible. Therefore, the technical problem this application aims to solve is to implement software support for non-standard quantization configurations on existing edge-side neural network hardware specifications, thereby overcoming the shortcomings of the existing quantization configuration schemes.

[0069] To facilitate understanding of the embodiments of this application, the following exemplary examples illustrate the scenarios in which the artificial intelligence processing device in this application is applied, which may include the following two scenarios.

[0070] Please see Figure 1A , Figure 1AThis is a schematic diagram of the structure of an intelligent question-answering system based on a large language model (LLM) according to an embodiment of this application. The system may include a training server 01, an application server 02, and a terminal device 03. The training server 01 can be a server cluster composed of multiple servers. The training server 01 can train an artificial intelligence model, such as a large language model (LLM), by executing model training methods and provide the trained LLM to the application server 02. The application server 02 may include the artificial intelligence processing device 10 in this embodiment, used to execute corresponding artificial intelligence processing methods (such as the LLM processing method described above). When a user asks a question through an AI application running on the terminal device 03, the trained LLM can be invoked by the artificial intelligence processing device 10 to provide an intelligent answer to the user's question. For example... Figure 1A The system can provide users with online question-and-answer services based on the LLM model. The terminal device 03 and the application server 02 are connected via network communication. Users can ask questions through the human-computer interaction interface of the AI ​​application running on the terminal device 03 (laptop, mobile phone, tablet, computer, smart vehicle, etc.). After receiving the user's question, the terminal device 03 sends the question to the application server 02. The application server 02 can call the artificial intelligence processing device 10 to execute LLM to provide the user with the corresponding answer based on the question raised by the user, and provide feedback to the user through the human-computer interaction interface of the AI ​​application.

[0071] Please see Figure 1B , Figure 1B This is a schematic diagram of another intelligent question-answering system based on a large language model provided in this application embodiment. The system may include a training server 01 and a terminal device 03, similar to the one described above. Figure 1A The difference in system architecture is that in this system architecture, the artificial intelligence processing device 10 is set in the terminal device 03. The execution of the AI ​​model and the application of AI can be realized simultaneously in the terminal device 03. That is, the terminal device 03 serves as both the execution device and the application device of the AI ​​model. When the user asks questions through the AI ​​application running in the terminal device 03, the terminal device 03 can call the trained large language model through the artificial intelligence processing device 10 to answer the user's questions intelligently.

[0072] It should be noted that the type of artificial intelligence model is not specifically limited in this embodiment; it can be any neural network model requiring model quantization or other AI models, which are not listed here. The terminal device 03 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. The training server 01 or application server 02 can be implemented by a separate server or a server cluster consisting of multiple servers.

[0073] Based on the above, please refer to Figure 2A , Figure 2A This is a schematic diagram of the structure of an artificial intelligence processing device provided in an embodiment of this application. The artificial intelligence processing device 10 (which can be simply referred to as the AI ​​processing device) specifically includes a data interface 100 and a processing unit 200. The data interface 100 can be used to receive relevant data for artificial intelligence processing, such as feature map matrices, and optionally, it also includes relevant configuration parameters (such as configuration parameters including multiple first-order width values). It is understood that the data interface 100 can be a physical interface or a physical + logical interface. Specifically, it can be a data transmission interface between the AI ​​processing device 10 and its coupled processor or on-chip system, etc. For example, an external coupling device (such as a processor) is responsible for distributing AI tasks and sending AI task-related data to the processing unit 200 through the data interface 100; while the processing unit 200 is used to execute and implement the relevant artificial intelligence processing methods in the embodiments of this application, specifically including online splitting of matrix element dimensions, matrix calculation, related vector calculation, concatenation, etc., of the received matrix, and further includes caching relevant initial data and intermediate data during the matrix calculation process.

[0074] The above Figure 2A The AI ​​processing device 10 shown can be located in any electronic device, such as a computer, mobile phone, tablet, smart wearable device, mobile device, etc. Specifically, the AI ​​processing device 10 can be applied in an AI inference system or device that has a pre-trained AI model deployed on it. For example, it can be located in the aforementioned... Figure 1A In the application server, or located in the above Figure 1BIn the terminal device, it can be located in a neural network accelerator that deploys a Large Language Model (LLM). The aforementioned artificial intelligence processing device 10 can also be a chip, chipset, or circuit board carrying a chip or chipset, which can operate under necessary software drivers. The AI ​​processing device 10 in this embodiment, deployed in relevant scenarios, can support non-standard quantization configurations through software implementation on top of the existing hardware specifications of edge-side neural network hardware acceleration units.

[0075] Furthermore, in one possible implementation, please see [link to relevant documentation]. Figure 2B , Figure 2B This is a schematic diagram of a processing unit in an artificial intelligence processing device provided in this application embodiment. The processing unit 200 specifically includes a vector calculation unit 210 and a matrix operation unit 220. Further, the matrix operation unit 220 internally includes a feature map cache 221, a weight cache 222, a matrix multiplication calculation unit 223, an accompanying calculation unit 224, and a general-purpose cache 225.

[0076] After receiving the feature map input through the data interface 100, the vector calculation unit 210 splits the feature map matrix (i.e., the first matrix) according to relevant parameters (such as multiple first-bit width values), and then sends the split sub-matrices (such as the first sub-matrices) to the feature map cache 221 (e.g., L0A) in the matrix operation unit 220 for caching. For example, the vector calculation unit 210 can split a first matrix with an element width of INT8 into a first sub-matrix with a high 4-bit width and a first sub-matrix with a low 4-bit width. Correspondingly, the weight cache 222 (e.g., L0B) can be used to cache the weight matrix (i.e., at least one second matrix) obtained from the general cache 225 (e.g., L1), and the matrix multiplication calculation unit 223 performs matrix multiplication calculations on the multiple first sub-matrices and at least one second matrix stored in the feature map cache 221 and the weight cache 222. For example, the high 4 bits of the first submatrix are multiplied by the second matrix to obtain the high 4 bits of the sub-result, and the low 4 bits of the first submatrix are multiplied by the second matrix to obtain the low 4 bits of the sub-result. Further, the matrix multiplication unit 223 inputs the high 4 bits and low 4 bits of the sub-result into the accompanying computation unit 224. The accompanying computation unit 224 can be used to perform quantization, sign processing, or sign addition processing (including sign addition processing for unsigned sub-results and bias term adjustment when necessary) on the multiple sub-results output by the matrix multiplication unit 223, and finally concatenate the processed sub-results to obtain the output result. This output result can be cached in a general-purpose cache 225 (e.g., L1) as the final result of the matrix operation, or it can be used as an intermediate result to continue participating in the next round of AI computation. Optionally, the process of concatenating the processed sub-results to obtain the output result can also be performed by the vector computation unit 210. It is understood that in Figure 2B In the processing unit 200, the vector calculation unit 210 and the matrix operation unit 220 can be logically divided, and the vector calculation unit 210 and the matrix operation unit 220 can be located on the same chip.

[0077] In another possible implementation, please see Figure 2C , Figure 2C This is another schematic diagram of the structure of the processing unit in the artificial intelligence processing device provided in this application embodiment. Figure 2C In the middle, the vector computing unit 210 can not only realize such as Figure 2B The functions described in the vector calculation unit 210 can also achieve, for example, Figure 2B The related functions of the on-path computing unit 224. That is, Figure 2C In the middle, through the vector computing unit 210, it covers Figure 2BThe vector computation unit 210 and the associated computation unit 224 in the matrix multiplication unit 223 perform the following functions: Specifically, this involves quantization, symbol processing, or bias processing on multiple sub-results obtained from the matrix multiplication unit 223. The vector computation unit 210 receives the multiple sub-results output from the matrix multiplication unit 223 and retrieves the decomposed inverse quantization and bias coefficients from the general-purpose cache 225 to perform the relevant processing on the multiple sub-results. Then, the vector computation unit 210 concatenates the multiple sub-results that have undergone the above quantization and bias processing and stores them in the general-purpose cache 225 as an intermediate result or as the final output result. For example, in... Figure 2C The process involves different output stages in terms of timing, including output ①, output ②, output ③, and output ④. Output ① refers to the output of the weight matrix, i.e., at least one second matrix, from the general cache 225 to the weight cache 222. Output ② refers to the process of calculating multiple sub-results in the matrix multiplication calculation unit 223 and then sending these sub-results to the general cache 225 for caching. The general cache 225 then sends the aforementioned sub-results, along with pre-cached inverse quantization coefficients (which can be split) and bias term coefficients (optional, for example, bias adjustment coefficients corresponding to the addition of signs to unsigned sub-results when needed), to the vector calculation unit 210 for computation. The vector calculation unit 210 performs correlation processing on the multiple sub-results based on the aforementioned correlation coefficients, and then concatenates the processed sub-results to generate an output result. The concatenated output result is then output to the general cache 225 (corresponding to output ③). If the output result is an intermediate result, it can be cached in the general cache 225. If the output result is the final output result, it can be output (corresponding to output ④).

[0078] When the artificial intelligence processing device in this application is applied to the field of neural networks, it can specifically be a neural network processing unit (NPU). Please refer to [link to relevant documentation]. Figure 2D , Figure 2D This is another hardware structure diagram of an artificial intelligence processing device provided in the embodiments of this application. Figure 2D In this context, the artificial intelligence processing device 10 is the neural network processor 10. For example... Figure 2DThe Neural Processing Unit (NPU) 10 shown is mounted as a coprocessor on the CPU 40, and tasks are allocated by the CPU 40. The NPU 10 includes a processing unit 200 and a data interface 100. The core of the processing unit 200 consists of an arithmetic circuit 303 and a vector operation unit 307. The controller 304 executes instructions from the instruction fetch memory 309 to control the arithmetic circuit 303 to retrieve matrix data from memory (such as input memory 301 and weight memory 302) and perform multiplication operations. It should be noted that the computing device or the device used to execute the artificial intelligence processing method in the embodiments of this application can be the aforementioned NPU 10, or it can be a device including the aforementioned NPU 10, CPU 40, and external memory 50. It is understood that... Figure 2D The vector operation unit 307 in the NPU 10 can correspond to the above. Figure 2B or Figure 2C The vector calculation unit 210, the arithmetic circuit 303, and the accumulator 308 in the above can correspond to the above. Figure 2B or Figure 2C The matrix multiplication calculation unit 223 in the middle.

[0079] Specifically, Figure 2D The processing unit 200 in the neural network processor 10 may specifically include an input memory 301, a weight memory 302, an arithmetic circuit 303, a controller 304, a Direct Memory Access Controller (DMAC) 305, a unified memory 306, an accumulator 308, a vector operation unit 307, an accumulator 308, and an instruction fetch memory 309. The data interface 100 may include a bus interface unit 310.

[0080] The input memory 301, weighted memory 302, unified memory 306, and instruction fetch memory 309 can all be on-chip memories. External memory 50 is proprietary to this NPU10 hardware architecture. For example, it could be memory coupled to CPU40.

[0081] In some implementations, the arithmetic circuit 303 may internally include multiple processing engines (PEs). In some implementations, the arithmetic circuit 303 is a two-dimensional pulsating array. Optionally, the arithmetic circuit 303 may also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor. Figure 2DAs shown, the arithmetic circuit 303 has two inputs, which are used to receive data sent from different memories (such as input memory 301 and weight memory 302) and multiply the two data; the accumulator 308 can be used to accumulate the calculation results after multiplying the matrix elements to obtain the accumulated calculation result.

[0082] For example, suppose there is an input feature map matrix A (such as multiple first sub-matrices after splitting in this application), a weight matrix B (such as at least one second matrix in this application), and an output matrix C (i.e., the intermediate output result or the final output result in this application). The arithmetic circuit 303 retrieves the corresponding data of matrix B from the weight memory 302 and caches it on each PE in the arithmetic circuit 303. The arithmetic circuit 303 retrieves the data of matrix A from the input memory 301 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 308.

[0083] The unified memory 306 can be used to store input data (e.g., the first matrix and at least one second matrix in this application) and output data (e.g., multiple sub-results, output results, etc. in this application). Weight data (e.g., at least one second matrix in this application) can be directly transferred to the weight memory 302 via the storage unit access controller (DMAC) 305. Input data can also be transferred to the unified memory 306 via the DMAC 305.

[0084] The Bus Interface Unit (BIU) 310 can be used for interaction between the bus and the DMAC 305 and the Instruction Fetch Buffer 309. For example, it can be used by the Instruction Fetch Buffer 309 to fetch instructions from external memory, and it can also be used by the memory access controller 305 to retrieve the original data of the input matrix A or weight matrix B, or related split data, bit width parameters, etc., from the external memory 50.

[0085] The Memory Access Controller (DMAC) 305 is mainly used to move input data from external memory 50 (such as DDR) to unified memory 306, or to weighted data to weighted memory 302, or to input data to input memory 301.

[0086] The vector operation unit 307 can further process the output of the operation circuit when needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector operation unit 307 is mainly used for computation in non-convolutional / FC layers of neural networks, such as pooling, batch normalization, and local response normalization. In some implementations, the vector operation unit 307 can store the processed output vector in the unified memory 306. For example, the vector operation unit 307 can apply nonlinear functions to the output of the operation circuit 303, such as the vector of accumulated values, to generate activation values. In some implementations, the vector operation unit 307 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as the activation input to the operation circuit 303, for example, for use in subsequent layers of the neural network.

[0087] Furthermore, the controller 304 is also connected to an instruction fetch buffer 309, which is used to store the instructions used by the controller 304.

[0088] For example, the main CPU 40 receives, as follows Figure 1A or Figure 1B The training server in the middle or Figure 1A The application server 02 sends the feature map input matrix, i.e., the first matrix, and stores it in external memory 50. Then, DMAC 305 moves the first matrix to unified memory 306 via bus interface unit 310. Similarly, at least one second matrix can be moved to unified memory in the same way. Then, DMAC 305 controls the movement of the first matrix from unified memory 306 to vector operation unit 307 for splitting. After the first matrix is ​​split into multiple second matrices by vector operation unit 307, they are sent to input memory 301 for caching. Simultaneously, DMAC... Unit 305 can move at least one second matrix obtained from external memory 50 to weight memory 302. At this point, the input data and weight data for neural network calculations are cached in input memory 301 and weight memory 302 respectively, ready for use. Under the control of the controller 304's control instructions (fetched from instruction fetch memory 309), the arithmetic circuit 303 and accumulator 308 can perform matrix element multiplication and accumulation calculations to obtain multiple sub-results. These sub-results can be temporarily stored in vector operation unit 307 for subsequent quantization processing, symbol processing, etc. Afterwards, vector operation unit 307 can also be used to concatenate the multiple sub-results to obtain the output result.

[0089] It is understandable that the above Figure 2A , Figure 2B , Figure 2C and Figure 2D The illustrated structure does not constitute a specific limitation on the artificial intelligence processing device 10. In other embodiments of this application, the artificial intelligence processing device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0090] Based on the above Figure 1A-Figure 1B System architecture and Figures 2A-2D The structure of any artificial intelligence processing device 10 in this application is described in detail, and the specific functions implemented by the artificial intelligence processing device 10 in this embodiment may include the following:

[0091] Data interface 100 is used to receive a first matrix, which is a feature map matrix;

[0092] The processing unit 200 is configured to split each matrix element in the first matrix into multiple matrix elements based on multiple first-width values ​​to obtain multiple first sub-matrices, wherein the size of each of the multiple first sub-matrices is the same as that of the first matrix; to perform calculations on each of the multiple first sub-matrices with each of at least one second matrix to obtain multiple sub-results, wherein the at least one second matrix is ​​a weight matrix; and to concatenate the multiple sub-results to obtain an output result.

[0093] Specifically, this application relates to the field of artificial intelligence (AI), primarily focusing on improvements to the computational process of AI hardware (such as the AI ​​processing device 10) during inference operations. Furthermore, neural network models are a crucial concept in AI learning, achieving the learning and processing of complex data through interconnected layers of neurons. During computation, weights and biases are key parameters, determining the connection strength and activation status between neurons. In practical applications, the computational process of a neural network can be converted into matrix form, thereby achieving efficient computation and optimization. Specifically, the input, weights, and biases can be expanded into matrices, and matrix multiplication and addition operations can be used to perform neural network computations. In this application embodiment, the improvement lies in the computational process between these matrices. The first matrix is ​​a feature map matrix, and at least one second matrix is ​​a weight matrix. When at least one second matrix is ​​a single second matrix, it indicates that the weight matrix has not been split; when at least one second matrix is ​​multiple second matrices, it indicates that the multiple second matrices are obtained by splitting the original weight matrix. It is understandable that, assuming the first matrix comprises M rows * K columns of elements, then the size of the at least one or more second matrices is always K rows * N columns. Furthermore, assuming the AI ​​processing device in this embodiment supports INT8-8 and INT4-4 from a hardware configuration perspective, then in this application, if the first matrix is ​​used as a feature map matrix before splitting, it may be of type INT16. That is, the hardware quantization configuration of the AI ​​processing device does not support the bit size of the matrix elements of the first matrix. Therefore, in this embodiment, the processing unit 200 uses multiple first-bit width values ​​to split each matrix element in the first matrix to obtain multiple sub-matrices with the same matrix size but a bit width smaller than the bit width of the matrix elements in the original first matrix. Optionally, the specific splitting process of the matrix elements can be based on the bit width value from high to low bits, or it can be based on the bit width value from low to high bits; this embodiment does not specifically limit this. Furthermore, during matrix calculation, the sub-matrices after splitting are calculated separately to obtain multiple sub-results. Since the feature map is uncertain, it can be split online in the AI ​​processing device 10. The weights, however, are determined during model transformation, so they can be split offline. That is, the first matrix can be split online, and the second matrix can be either a pre-split weight matrix (i.e., split offline) or the original, unsplit weight matrix. Optionally, model transformation and / or weight splitting can be performed as described above. Figure 1A or Figure 1BThe calculation can be performed on training server 01 or application server 02, or on terminal device 03 (such as a processor). It is understood that the first matrix and at least one second matrix can be part or all of the matrices used in a particular AI computation.

[0094] Since training scenarios typically use floating-point data, while this application is applicable to integer-type scenarios, the embodiments of this application are suitable for quantized neural networks and can be applied to AI inference scenarios, i.e., AI application scenarios, specifically quantized inference scenarios. The embodiments of this application can split the bit width of each element in the feature map and / or weight matrix. Typically, NPUs only support operations with fixed bit widths after leaving the factory. For example, existing NPUs usually support INT16*8, INT8*8, or INT4*4, and the format is fixed. However, the embodiments of this application can flexibly split and concatenate data types, allowing the NPU to support improved precision or computational efficiency without hardware changes, enabling scenario diversification and overcoming the data format bottleneck of weights or feature maps.

[0095] This application provides an artificial intelligence (AI) processing device applied in AI inference scenarios. By performing online splitting of matrix elements in the feature map matrix during matrix operations, the model quantization configuration is adapted to the hardware bit quantization configuration of the AI ​​processing device. This achieves software support for non-standard quantization configurations on existing edge AI hardware specifications. Specifically, this application design provides an AI processing device that can split matrix elements online based on configuration parameters (i.e., multiple first-bit width values), thereby enabling arbitrary combinations of quantization bit numbers for the feature map matrix. This solves the technical problem that existing AI hardware typically only provides fixed quantization configurations for users. For example, some AI hardware supports INT8-8 and INT4-4 quantization, but the feature map and weights to be calculated are INT16 and INT8 respectively. With the splitting method described in this application, the INT16 feature map matrix can be split into two INT8 matrices, adapting to the hardware's bit quantization configuration and enabling flexible combinations of feature map and weight quantization bit numbers. In summary, the embodiments of this application provide an artificial intelligence processing device for supporting a flexible quantization computing scheme with flexible quantization configuration. It enables the model to be deployed on the edge and reconstructed with a quantization configuration that is friendly to the AI ​​model (such as an LLM model) through the quantization configuration supported by the hardware. That is, without changing the hardware, on the basis of the fixed quantization configuration provided by the hardware, the software adds support for quantization strategies. In some application scenarios, this can reduce bandwidth and power consumption and improve computing speed. For example, when applied to LLM scenarios, it can reduce the ROM and RAM usage of LLM and improve the inference performance of LLM.

[0096] For example, the process for elastic quantization calculation of the feature map matrix is ​​as follows:

[0097] Step 1: Take the given feature map as input, and then... Figure 2A Processing unit 200 in or Figure 2B or Figure 2C Vector computation unit 210 in, or Figure 2D The vector operation unit 307 in the neural network splits each element into two or more high-order and low-order elements. Since the feature map is computed online in the neural network, this splitting step can be completed online.

[0098] Step 2: Through the above Figure 2A Processing unit 200 in or Figure 2B or Figure 2C Matrix multiplication calculation unit 223 in the middle, or Figure 2DThe computation circuit 303 in the middle performs operations on the multiple sub-feature maps obtained by splitting them with the weights respectively to obtain multiple corresponding sub-results (outputs).

[0099] Step 3: Through the above Figure 2A Processing unit 200 in or Figure 2B The accompanying calculation unit 224 or vector calculation unit 210, or Figure 2C Vector computation unit 210 in, or Figure 2D The vector operation unit 307 in the middle merges multiple sub-outputs into the final output according to the splitting method.

[0100] In one possible implementation, the processor (CPU) 40 coupled to the artificial intelligence processing device 10 can be used to: receive a configuration file, determine first configuration parameters based on the configuration file, and send the first configuration parameters to the processing device 10 (e.g., an NPU). Further, the data interface 100 in the artificial intelligence processing device 10 is also used to: receive the first configuration parameters, the first configuration parameters including the plurality of first-bit width values. Specifically, the configuration file can be... Figure 1A or Figure 1B Generated on training server 01 in the middle, Figure 1A Application server 02 or Figure 1B After receiving the configuration file, the processor in the terminal device 03 determines the first configuration parameters mentioned above and then sends them to the AI ​​processing device 10. It should be noted that the data interface 100 mentioned above may include physical interfaces or logical interfaces. For example, it may include different logical interfaces located on the same physical interface in the AI ​​processing device 10, or it may include different physical interfaces distributed in the AI ​​processing device 10. No specific limitation is made here.

[0101] In one possible implementation, the plurality of first bit width values ​​are A1, A2, ..., Ax, where x is an integer greater than 1. The processing unit is specifically used to: split each matrix element in the first matrix into x matrix elements with bit widths of A1, A2, ..., Ax, to obtain x first sub-matrices with the same size as the first matrix but different bit widths. Specifically, the specific splitting process of the matrix elements can be based on the bit width values ​​and split in a certain order, such as from high to low or from low to high. The bit width of the matrix elements in the first matrix is ​​A, the plurality of bit width values ​​are A1, A2, ..., Ax, where x is an integer greater than 1, and the sum of A1, A2, ..., Ax is A.

[0102] Please see Figure 3A , Figure 3AThis application provides a schematic diagram of splitting a first matrix into multiple first sub-matrices. In this application embodiment, the processing unit 200 can split each matrix element in the feature map matrix separately, specifically as follows: Figure 3A As shown, assuming the first matrix is ​​an M x K matrix with each element having a bit width of A, the processing unit 200 splits each element in the first matrix into x sub-elements with bit widths A1, A2, ..., Ax, to generate x first sub-matrices of the same size as the first matrix but with different bit widths, where A1 + A2 + ... + Ax = A. For example, for the element A in the first row and first column of the first matrix... 11， Arrange A in order from left to right 11 It is split into matrix elements with bit widths of A1, A2...Ax, respectively A1 11 A2 11 ......Ax 11 For matrix element A in the second row and first column 21 Arrange A in order from left to right. 21 It is split into matrix elements with bit widths of A1, A2...Ax, respectively A1 21 A2 21 ......Ax 21 And so on, for the matrix element A in the Mth row and Kth column... MK Arrange A in order from left to right. MK It is split into matrix elements with bit widths of A1, A2...Ax, respectively A1 MK A2 MK ......Ax MK Finally, according to the original positions of the matrix elements in the first matrix, the M rows * K columns of element A1 with a bit width of A1 are... 11 A1 21 A1 31 ......A1 Mk As Figure 3A The first submatrix shown contains M rows * K columns of element A2. 11 A2 21 A2 31 ......A1 Mk As such Figure 3A The second submatrix shown, and so on, uses M rows * K columns of elements A1 of width Ax. MK A2 MK ......Ax MK As such Figure 3A The xth first submatrix shown in the figure is used to obtain x first submatrices in the end.

[0103] In one possible implementation, the at least one second matrix comprises y second matrices obtained by splitting each matrix element of the third matrix into y matrix elements, wherein each of the y second matrices has the same size as the third matrix, and y is an integer greater than 1. Specifically, the at least one second matrix calculated by the accelerator is the y second matrices obtained by splitting the original third matrix. In this embodiment, the at least one second matrix (i.e., the y second matrices) is also obtained by splitting the original weight matrix (i.e., the third matrix) by matrix elements, similar to the splitting method used for the first matrix. For example, the y second matrices can be pre-split offline or online in the AI ​​processing device. That is, in the matrix operation process of this embodiment, not only can the bit width of the matrix elements of the feature map matrix be split, but further, the bit width of the matrix elements of the weight matrix can also be split. In this way, a more flexible combination of quantization bits for the feature map and weight can be achieved.

[0104] In one possible implementation, the processing unit is further configured to: based on multiple second bit-width values, split each matrix element in the third matrix into y matrix elements with bit widths of B1, B2, ..., By, to obtain y second matrices with the same size as the third matrix but different bit widths. The processing unit in the accelerator performs this online splitting. The bit width of the matrix elements in the third matrix is ​​B, and the bit widths of the matrix elements in the y second sub-matrices are B1, B2, ..., By, where y is an integer greater than 1, and the sum of B1, B2, ..., By is B. In this embodiment, the processing unit in the AI ​​processing device can further perform online splitting of the third matrix into y second matrices, that is, splitting the bit width of the matrix elements from two dimensions: the feature map matrix and the weight matrix, thereby supporting more model quantization strategies through software without changing the AI ​​acceleration hardware.

[0105] In another possible implementation, the processing unit 200 receives the y second matrices, which are obtained by the processor (CPU) 40 after splitting each matrix element in the third matrix into multiple matrix elements based on multiple second bit-width values. That is, in this embodiment, the splitting of the third matrix can be performed online on the processor, and then the processor sends it to the processing unit 200 in the AI ​​processing device 10 through relevant coupling devices (such as data interfaces).

[0106] Please see Figure 3B , Figure 3BThis application provides a schematic diagram of splitting a third matrix into multiple second matrices according to an embodiment of the present application. In this embodiment, the processing unit 200 can also split each matrix element in the weight matrix separately, as shown below. Figure 3A As shown, assuming the third matrix is ​​a K x N matrix with each element having a bit width of B, the processing unit 200 splits each element in the third matrix into y elements with bit widths of B1, B2, ..., By, to generate y second matrices of the same size as the third matrix but with different bit widths, where B1 + B2 + ... + By = B. For example, for the element B in the first row and first column of the second matrix... 11， Arrange A in order from left to right 11 It is split into matrix elements with bit widths of B1, B2...Bx, respectively B1 11 B2 11 ......Bx 11 For matrix element A in the second row and first column 21 Arrange B in order from left to right. 21 It is split into matrix elements with bit widths of B1, B2...By, respectively B1 21 B2 21 ......Bx 21 And so on, for the matrix element B in the Mth row and Kth column... MK Arrange B in order from left to right. MK It is split into matrix elements with bit widths of B1, B2...Bx, respectively B1 MK B2 MK ......Bx MK Finally, according to the original positions of the matrix elements in the third matrix, the K rows * N columns of element B1 with a bit width of B1 are... 11 B1 21 B1 31 ......B1 Mk As Figure 3B The first second matrix shown contains K rows * N columns of element B2. 11 B2 21 B2 31 ......B1 Mk As such Figure 3A The second matrix shown, and so on, uses K rows * N columns of elements A1 with a bit width of Ax. MK A2 MK ......Ax MK As such Figure 3A The x-th second matrix shown in the figure eventually yields y second matrices.

[0107] For example, the process for calculating the elastic quantization of the weight matrix is ​​as follows:

[0108] Step 1: Split each element in the weights of the neural network into two or more high-order and low-order elements. Since the weights of the neural network are fixed offline, this step can be done offline. That is, this step can be performed during the process described above. Figure 1A or Figure 1B The offline splitting was completed in training server 01.

[0109] Step 2: Through the above Figure 2A Processing unit 200 in or Figure 2B or Figure 2C Matrix multiplication calculation unit 223 in the middle, or Figure 2D The computation circuit 303 in the middle performs operations on the multiple sub-weights obtained from the splitting with the feature map to obtain the corresponding multiple sub-outputs.

[0110] Step 3: Through the above Figure 2A Processing unit 200 in or Figure 2B The accompanying calculation unit 224 or vector calculation unit 210, or Figure 2C Vector computation unit 210 in, or Figure 2D The vector operation unit 307 in the middle merges multiple sub-outputs into the final output according to the splitting method.

[0111] In one possible implementation, the processing unit is specifically used to: perform matrix operations on each of the x first sub-matrices and each of the y second matrices to obtain x*y sub-results. For details, please refer to [link to relevant documentation]. Figure 4A , Figure 4A This is a schematic diagram illustrating how multiple first sub-matrices and multiple second matrices are calculated to obtain multiple sub-results, as provided in an embodiment of this application. Figure 4A As shown, assuming the first matrix is ​​split into x first sub-matrices and the third matrix is ​​split into y second matrices, then each of the x first sub-matrices is subjected to a matrix operation with each of the y second sub-matrices; after x*y rounds of matrix operations, x*y sub-results are obtained, A1*B1, A2*B1, ..., Ax*B1; This also includes... Figure 4A The sub-results A1*B2, A2*B2, ..., Ax*B2 (not shown in 4C) and A1*By, A2*By, ..., Ax*By (shown in 4C) comprise a total of x*y sub-results.

[0112] In one possible implementation, the processing unit 200 is further configured to: obtain the split quantization coefficients, the split quantization coefficients including multiple sub-quantization coefficients; and perform quantization operations on each of the multiple sub-results with one of the multiple sub-quantization coefficients to obtain the multiple sub-results after the quantization operation. Specifically, one sub-result corresponds to one quantization coefficient, that is, one sub-result is operated on with a corresponding quantization coefficient. Since the process of fusing multiple sub-results may involve the calculation of correlation coefficients, these coefficients are used to perform correlation processing on the multiple sub-results, and finally, the multiple processed sub-results are fused to obtain the output result. In this embodiment, the quantization coefficients involved in the matrix operation process can also be adaptively split according to the rules of splitting the feature map matrix or weight matrix to adapt to the quantization configuration supported by the AI ​​processing device. Specifically, each of the multiple sub-results obtained after the operation of the feature map matrix and the weight matrix is ​​processed with quantization coefficients, that is, one sub-result corresponds to one quantization coefficient, and finally, the multiple quantized sub-results are calculated.

[0113] In one possible implementation, the data type of the first matrix is ​​an unsigned data type; the processing unit 200 is further configured to: perform sign addition processing on the unsigned sub-results among the plurality of sub-results after the quantization operation, respectively, to obtain the plurality of sub-results after the processing. Specifically, after performing matrix multiplication, the unsigned data type can be converted to a signed data type for operation, specifically by adjusting the model parameters (bias term) to ensure the correctness of the calculation result. Specifically, in some scenarios, if the first matrix is ​​originally signed data, the higher-order matrix elements after splitting may already carry signed bits. Operations on these signed matrix elements are signed data type operations, while operations on the lower-order matrix elements without signed bits are unsigned data type operations. If, due to computational or scenario requirements, the operation type needs to be converted to a signed data type, then the unsigned sub-results in the multiple sub-results after quantization processing need to be signed to convert the operation type to a signed type. Conversely, if the first matrix is ​​originally unsigned data, then operations on the matrix elements after splitting the unsigned first matrix are all unsigned data type operations. Therefore, if, due to computational or scenario requirements, the operation type needs to be converted to a signed data type, then all unsigned sub-results need to be signed. In this embodiment, when the multiple sub-results after quantization include one or more unsigned sub-results, and in some computational scenarios, the operation type needs to be converted to a signed data type, then the one or more unsigned sub-results can be signed. That is, the AI ​​processing device in this application embodiment can quantize the multiple sub-results obtained in the calculation process of multiple first sub-matrices and at least one second matrix, then add signs to the unsigned sub-results, and then adjust them through a bias term to ensure the correctness of the calculation results. In this way, it can support the conversion of the operation process of unsigned data type to the operation of signed data type, so that more types of operations can be supported without changing the AI ​​hardware.

[0114] In one possible implementation, the matrix elements in the first matrix are either positive or negative integers. This application addresses the scenario of integer data, not floating-point data. The quantization strategy in this application's embodiments is applicable to scenarios where the matrix is ​​an integer or a negative number; that is, regardless of whether the scenario involves positive or negative integers, the artificial intelligence processing device in this application can achieve a more flexible combination of quantization bits.

[0115] This application describes the method of reconstructing the INT8-4 quantization configuration using the INT4-4 quantization processing unit. It is understood that this application is not limited to the scenario of reconstructing INT8-4 using INT4-4, but can be generalized to other quantization configurations, such as reconstructing INT16-8, INT16-4, etc. using the INT4-4 quantization processing unit.

[0116] The following diagram illustrates an INT8-4 multiplication, where the feature map (the first matrix) is 8 bits and the weight (the second matrix) is 4 bits, both being signed fixed-point numbers. The 8 bits of the feature map can be represented by two 4-bit numbers: the high 4 bits and the low 4 bits (the two first submatrices). Based on the values ​​in the feature map, we can distinguish between positive and negative scenarios:

[0117] Please see Figure 4B , Figure 4B This application provides an embodiment of an element decomposition diagram for a scene with a positive number feature map, as shown below. Figure 4B As shown, assuming a positive number scenario, the high 4 bits of a matrix element in the feature map matrix are 0011 and the low 4 bits are 1110, where the first 0 in the high 4 bits represents a positive number. The weight matrix is ​​0100, meaning the first matrix is ​​split into two first sub-matrices, and at least one second matrix is ​​a single second matrix. In the positive number scenario, INT8 with a sign is equal to INT4. 高4位-有符号 *16+INT4 低4位-无符号 ,like Figure 4B The INT8 value represents 62, with the high 4 bits being a signed number of 3 and the low 4 bits being an unsigned number of 14. Since 62 = 3 * 16 + 14, according to the associative law:

[0118] INT8 有符号 *Wint4 = INT4 高4位-有符号 *Wint4*16+INT4 低4位-无符号 *Wint4

[0119] Please see Figure 4C , Figure 4CThis application provides an embodiment of element splitting for a negative number scenario in the feature map. Assuming that in the negative number scenario, the high 4 bits of a matrix element in the feature map matrix are 1101 and the low 4 bits are 1010, where the first 1 in the high 4 bits represents a negative number, and the weight matrix is ​​0100, meaning the first matrix is ​​split into two first sub-matrices, and at least one second matrix is ​​a single second matrix. In the negative number scenario: since the negative numbers of signed fixed-point numbers are stored in two's complement form in processing unit 200, the negative number scenario needs to be analyzed separately. The value of a negative number in the original code can be described as: after removing the sign bit, the remaining values ​​are directly summed, and then the sign bit is used to invert the result. When calculating the two's complement, the sign bit is ignored first, resulting in a positive value, which is then subtracted from 2w-1 (to obtain the two's complement), and then the sign bit is used to invert the result. Here, w represents the number of bits. In the INT8-4 multiplication scenario, w = 8.

[0120]

[0121] For the two's complement of the high 4 bits, it is represented as therefore,

[0122]

[0123] in This represents the two's complement of the high 4 bits. This represents the lower 4 bits of the original code.

[0124] Considering that the high 4 bits need to be shifted left by 4 bits during the concatenation process, in the case of negative two's complement, an INT8 number can also be represented as:

[0125] INT8 有符号-补码 *Wint4 = INT4 高4位-有符号-补码 *Wint4*16+INT4 低4位-无符号-原码 *Wint4

[0126] Based on the above analysis, this application embodiment transforms an INT8-4 multiplication operation into two INT4-4 multiplication operations.

[0127] Furthermore, if the hardware unit (i.e., the AI ​​processing device 10) only supports signed fixed-point number operations, unsigned fixed-point numbers can also be converted to signed fixed-point number operations in the following manner:

[0128] INT4 无符号 *Wint4 = (INT4) 无符号 -8)*Wint4+8*Wint4

[0129] Among them, 8*Wint4 can be calculated offline as the bias, while (INT4) 无符号 -8) is a signed 4-bit integer.

[0130] This application provides a scheme for implementing elastic quantization of feature maps. The specific process of implementing this scheme using an NPU is described below, taking an INT8-4 implementation as an example to illustrate the calculation process of elastic quantization of feature maps. Please refer to... Figure 5A , Figure 5A This application provides a schematic diagram of the overall process for splitting and calculating a feature map matrix, and the overall calculation process specifically includes:

[0131] (1) Input Fixed-Point Activation Value Splitting: Since the input fixed-point activation values, i.e., the feature map matrix (input_int8), are calculated online during neural network inference, the first step is to split the int8 fixed-point activation values ​​into two sets of int4 fixed-point activation values. That is, the first matrix of int8 is split into two first sub-matrices of int4. These are named input_h4 and input_l4. The calculations are as follows:

[0132] input h4 =cast<int4_t> (input int8 &0xf0)>>4

[0133] input l4 =cast<int4_t> (input int8 &0x0f)-8

[0134] input in this process h4 input l4 It can be found in the above Figure 2A or Figure 2B Vector computation unit 210 in, or Figure 2D It is calculated in the vector operation unit 307 (NPU Vector Core).

[0135] (2) Calculate the inverse quantization coefficients of the two sets of 4-bit fixed-point matrix multiplication: Assume the quantization parameter of the input is q. input The quantization parameter corresponding to weight is q. w The inverse quantization coefficients of the two sets of fixed-point matrix multiplications are as follows:

[0136]

[0137]

[0138] This process can be achieved through offline weight transformation. For example, the two first sub-matrices input after the above split... h4 input l4 The corresponding qinput and the q corresponding to the weight matrix (i.e., the second matrix) w , can Figure 1A or Figure 1B The data was obtained from training server 01 and sent to it. Figure 1A Application server 02 or Figure 1B The terminal device 03 in the middle stores the data.

[0139] (3) Calculate input l4 Calculated bias l4 bias l4 =8*q w This process can be achieved through offline weight transformation, for example, in Figure 1A or Figure 1B The data was obtained from training server 01 and sent to it. Figure 1A Application server 02 or Figure 1B The terminal device 03 in the middle stores the data.

[0140] (4) Combine the results of the two fixed-point matrix multiplication operations:

[0141] output h4 =GEMM(input) h4 (weight)

[0142] output l4 =GEMM(input) l4 (weight)

[0143] output = DEQ(output) h4 deq_scale h4 )+DEQ(output l4 deq_scale l4 )+bias l4

[0144] This process can be achieved through the above... Figure 2A Processing unit 200 in or Figure 2B The accompanying calculation unit 224 or vector calculation unit 210, or Figure 2C Vector computation unit 210 in, or Figure 2D The calculation is completed using the vector operation unit 307 in the middle.

[0145] The above Figure 5A This describes the computational process of splitting the feature map separately and then calculating multiple sub-results with the weight matrix, which is the process described above. Figure 5AIn one matrix operation, the first matrix is ​​split online. The weight matrix can be split or not. For details on splitting the weight matrix, please refer to the following sections. Figure 5B The relevant description of the calculation process for the split weight matrix is ​​not repeated here.

[0146] This application provides a scheme for implementing elastic weight quantization. The calculation process of elastic weight quantization is described below using INT4-8 quantization as an example. It is understood that this application is not limited to the scenario of reconstructing INT4-8 from INT4-4, and can be generalized to other quantization configurations, such as INT8-16, INT4-16, etc. The principle of splitting the INT8 weight matrix into two signed INT4 matrices is the same as the principle of splitting the feature map matrix described above, and will not be repeated here. When performing bit splitting of the weight, please refer to [link to relevant documentation]. Figure 5B , Figure 5B This application provides a schematic diagram of the overall process for splitting and calculating a weight matrix, and the overall calculation process specifically includes:

[0147] (1) Fixed-point weighted splitting:

[0148] Since the input fixed-point weight values ​​(weight_int8) can be calculated offline, the first step is to split the int8 fixed-point activation values ​​offline into two groups of int4 fixed-point activation values. That is, to split the third matrix of the int8 into two second matrices of int4. These are named weight_h4 and weight_l4, and the specific calculation processes are as follows:

[0149] weight h4 =cast<int4_t> (weight int8 &0xf0)>>4

[0150] weight l4 =cast<int4_t> (weight int8 &0x0f)-8

[0151] The splitting of weight_h4 and weight_l4 in this process can be performed on the training side of the neural network, such as... Figure 1A or Figure 1B It can be executed on the training server 01 or the application server 02, or alternatively, on the inference or application side of the neural network. Figure 2D It is executed on the general-purpose processor (CPU) 40 of the terminal device.

[0152] (2) Calculate the inverse quantization coefficients of the two sets of 4-bit fixed-point matrix multiplication: Assume the quantization parameter of the input is q. input The quantization parameter corresponding to weight is q. w The inverse quantization coefficients of the two sets of fixed-point matrix multiplications are as follows:

[0153]

[0154]

[0155] The above process can also be obtained through offline weight transformation. For example, the two first sub-matrices input after the above split... h4 input l4 The corresponding q input And the two second matrices weight_h4, weight_l4, respectively, corresponding to q w , can Figure 1A or Figure 1B The data was obtained from training server 01 and sent to it. Figure 1A Application server 02 or Figure 1B The terminal device 03 in the middle stores the data.

[0156] (3) Combine the results of the three fixed-point matrix multiplication operations:

[0157] output h4 =GEMM(input, weight) h4 )

[0158] output l4 =GEMM(input, weight) l4 )

[0159] bias_term = GEMM(input, -8)

[0160] output = DEQ(output) h4 deq_scale h4 )+DEQ(output l4 deq_scale l4 -DEQ(bias_term, deq_scale) l4 )

[0161] This process can be achieved through the above. Figure 2A Processing unit 200 in or Figure 2B The accompanying calculation unit 224 or vector calculation unit 210, or Figure 2C Vector computation unit 210 in, or Figure 2D The calculation is completed using the vector operation unit 307 in the middle.

[0162] The above Figure 5B This describes the computational process of separately splitting the weights and then calculating multiple sub-results with the feature map matrix. It can be understood that the above... Figure 5B The text discusses the splitting of weights and the aforementioned methods. Figure 5A The feature map matrix is ​​split, and the splits can be combined. That is, in a single matrix operation, the online splitting of the first matrix and the offline or online splitting of the third matrix are involved at the same time. Specifically, the multiple first sub-matrices obtained by online splitting are combined with each of the multiple second matrices obtained by offline or online splitting to perform matrix calculations, resulting in multiple sub-results, which will not be elaborated here.

[0163] This application's embodiments enable the addition of quantization strategy support through software without changing the hardware computing unit. For example, it supports INT8-4 and INT16-8 quantization in neural network processors, while also enabling non-standard quantization bit counts, such as INT12-4 quantization. For scenarios involving feature map and weight alignment, it reduces the overhead of weight rearrangement; for instance, INT16-8 quantization requires sparse weight arrangement aligned to 16 bits. This application's embodiments achieve compact weight arrangement, reducing bandwidth and read / write power consumption by half. For example, experimental verifications have shown that for LLM, implementing this application's embodiments can achieve approximately 50% speedup and reduce NPU power consumption, enhancing the product's competitiveness in large-model scenarios.

[0164] Please see Figure 6 , Figure 6 This is a flowchart illustrating an artificial intelligence processing method provided in an embodiment of this application. This artificial intelligence processing method can be applied to artificial intelligence processing devices, neural network processors, or computing devices, etc. The method may include the following steps S601-S604, wherein...

[0165] Step S601: Receive the first matrix, which is a feature map matrix.

[0166] Step S602: Based on multiple first-order width values, split each matrix element in the first matrix into multiple matrix elements to obtain multiple first sub-matrices, the size of each of the multiple first sub-matrices being the same as the size of the first matrix.

[0167] Step S603: Each of the plurality of first sub-matrices is calculated with each of the at least one second matrix to obtain a plurality of sub-results, wherein the at least one second matrix is ​​a weight matrix.

[0168] Step S604: Concatenate the multiple sub-results to obtain the output result.

[0169] In one possible implementation, the plurality of first-order width values ​​are A1, A2...Ax, where x is an integer greater than 1;

[0170] The step of calculating each of the plurality of first sub-matrices with each of at least one second matrix to obtain a plurality of sub-results includes:

[0171] Each element in the first matrix is ​​split into x matrix elements with bit widths A1, A2, ..., Ax to obtain x first submatrices with the same size as the first matrix but different bit widths.

[0172] In one possible implementation, the at least one second matrix comprises y second matrices obtained by splitting each matrix element of the third matrix into y matrix elements, wherein each of the y second matrices has the same size as the third matrix, and y is an integer greater than 1.

[0173] In one possible implementation, the method further includes: splitting each matrix element in the third matrix into y matrix elements with bit widths of B1, B2...By based on multiple second bit width values, so as to obtain y second matrices with the same size as the third matrix but different bit widths.

[0174] In one possible implementation, the method further includes:

[0175] Obtain the split quantization coefficients, wherein the split quantization coefficients include multiple sub-quantization coefficients;

[0176] Each of the multiple sub-results is quantized with one of the multiple sub-quantization coefficients to obtain the multiple sub-results after the quantization operation.

[0177] In one possible implementation, the method further includes: performing sign addition processing on the unsigned sub-results among the plurality of sub-results after the quantization operation, so as to obtain the plurality of sub-results after the processing.

[0178] In one possible implementation, the matrix elements in the first matrix are either positive integers or negative integers.

[0179] It should be noted that the specific process of the artificial intelligence processing method described in the embodiments of this application can be found in the above. Figures 1A-5B The relevant descriptions in the application embodiments described herein will not be repeated here.

[0180] This application also provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, which, when executed by a computing device, includes some or all of the steps described in any of the above method embodiments.

[0181] This application also provides a computer program product including instructions that, when executed by a computing device, enable the computing device to perform some or all of the steps of any artificial intelligence processing method.

[0182] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0183] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0184] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0185] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0186] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0187] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium may include various media capable of storing program code, such as a USB flash drive, portable hard drive, magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0188] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An artificial intelligence processing device, characterized in that, The device includes a data interface and a processing unit; The data interface is used to receive a first matrix, which is a feature map matrix; The processing unit is used for: Based on multiple first-order width values, each matrix element in the first matrix is ​​split into multiple matrix elements to obtain multiple first sub-matrices, the size of each of the multiple first sub-matrices being the same as the size of the first matrix; Each of the plurality of first sub-matrices is calculated with each of the at least one second matrix to obtain a plurality of sub-results, wherein the at least one second matrix is ​​a weight matrix; The multiple sub-results are concatenated to obtain the output result.

2. The apparatus according to claim 1, characterized in that, The plurality of first-order width values ​​are A1, A2...Ax, where x is an integer greater than 1; The processing unit is specifically used to: split each matrix element in the first matrix into x matrix elements with bit widths of A1, A2...Ax, so as to obtain x first sub-matrices with the same size as the first matrix but different bit widths.

3. The apparatus according to claim 1 or 2, characterized in that, The at least one second matrix includes y second matrices obtained by splitting each matrix element of the third matrix into y matrix elements, wherein each of the y second matrices has the same size as the third matrix, and y is an integer greater than 1.

4. The apparatus according to claim 3, characterized in that, The processing unit is further configured to: based on multiple second bit width values, split each matrix element in the third matrix into y matrix elements with bit widths of B1, B2...By, so as to obtain y second matrices with the same size as the third matrix but different bit widths.

5. The apparatus according to any one of claims 1-4, characterized in that, The processing unit is further configured to: Obtain the split quantization coefficients, wherein the split quantization coefficients include multiple sub-quantization coefficients; Each of the multiple sub-results is quantized with one of the multiple sub-quantization coefficients to obtain the multiple sub-results after the quantization operation.

6. The apparatus according to claim 5, characterized in that, The processing unit is further configured to: perform sign addition processing on the unsigned sub-results among the multiple sub-results after the quantization operation, so as to obtain the multiple sub-results after the processing.

7. The apparatus according to any one of claims 1-6, characterized in that, The matrix elements in the first matrix are either positive integers or negative integers.

8. An artificial intelligence processing method, characterized in that, The method includes; Receive the first matrix, which is a feature map matrix; Based on multiple first-order width values, each matrix element in the first matrix is ​​split into multiple matrix elements to obtain multiple first sub-matrices, the size of each of the multiple first sub-matrices being the same as the size of the first matrix; Each of the plurality of first sub-matrices is calculated with each of the at least one second matrix to obtain a plurality of sub-results, wherein the at least one second matrix is ​​a weight matrix; The multiple sub-results are concatenated to obtain the output result.

9. The method according to claim 8, characterized in that, The plurality of first-order width values ​​are A1, A2...Ax, where x is an integer greater than 1; The step of calculating each of the plurality of first sub-matrices with each of at least one second matrix to obtain a plurality of sub-results includes: Each matrix element in the first matrix is ​​split into x matrix elements with bit widths A1, A2, ..., Ax to obtain x first submatrices with the same size as the first matrix but different bit widths.

10. The method according to claim 8 or 9, characterized in that, The at least one second matrix includes y second matrices obtained by splitting each matrix element of the third matrix into y matrix elements, wherein each of the y second matrices has the same size as the third matrix, and y is an integer greater than 1.

11. The method according to claim 10, characterized in that, The method further includes: based on multiple second bit width values, splitting each matrix element in the third matrix into y matrix elements with bit widths of B1, B2...By, to obtain y second matrices with the same size as the third matrix but different bit widths.

12. The method according to any one of claims 8-11, characterized in that, The method further includes: Obtain the split quantization coefficients, wherein the split quantization coefficients include multiple sub-quantization coefficients; Each of the multiple sub-results is quantized with one of the multiple sub-quantization coefficients to obtain the multiple sub-results after the quantization operation.

13. The method according to claim 12, characterized in that, The method further includes: performing sign addition processing on the unsigned sub-results among the multiple sub-results after the quantization operation, so as to obtain the multiple sub-results after the processing.

14. The method according to any one of claims 8-13, characterized in that, The matrix elements in the first matrix are either positive integers or negative integers.

15. A computing device, characterized in that, It includes the artificial intelligence processing device and processor as described in any one of claims 1-7, wherein the processor is coupled to the artificial intelligence processing device.

16. The computing device according to claim 15, characterized in that, The processor is further configured to: based on multiple second bit width values, split each matrix element in the third matrix into y matrix elements with bit widths of B1, B2...By, to obtain y second matrices with the same size as the third matrix but different bit widths.

17. A computer-readable storage medium, characterized in that, The computer-readable medium is used to store program code, which, when executed by a computing device, implements the method described in any one of claims 8-14.

18. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 8-14.

Images