Training method and device applied to neural network and related product

Abstract

The application relates to a training method and device applied to a neural network and related products. The application relates to a board card which comprises a storage device, an interface device, a control device and an artificial intelligence chip; wherein the artificial intelligence chip is connected with the storage device, the control device and the interface device respectively; the storage device is used for storing data; the interface device is used for realizing data transmission between the artificial intelligence chip and an external device; and the control device is used for monitoring the state of the artificial intelligence chip. The board card can be used for executing artificial intelligence operation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training method and apparatus for neural networks, as well as related products. Background Technology

[0002] With the continuous development of neural network technology, its application fields are becoming increasingly widespread, finding excellent applications in image recognition, speech recognition, natural language processing, and other areas. Neural networks need to be trained to obtain their structural parameters before being used in various fields for inference operations. The neural network training process involves inputting a sufficient number of samples into the network and adjusting its structure (mainly adjusting the weights) using a certain algorithm to make the network's output match the expected value. These structural parameters are then applied to the neural network's inference operations, which is the actual process of using the neural network.

[0003] Neural networks include various computational layers such as convolutional layers and fully connected layers, each requiring large amounts of data to execute complex algorithms. During neural network training, single-precision floating-point numbers (FP32) are often used. However, due to the complexity of neural network algorithms and the excessive bit length of FP32 floating-point numbers, the training process consumes excessive storage space, suffers significant computational latency, and has low hardware efficiency. Therefore, how to improve training efficiency, reduce storage consumption, and enhance hardware performance while maintaining neural network training accuracy is a pressing issue in the field of neural network technology. Summary of the Invention

[0004] In view of this, the present invention proposes a training method and apparatus for neural networks, as well as related products.

[0005] According to one aspect of the present invention, a processor for performing neural network training is provided, the neural network training comprising three stages: forward operation, backward operation, and weight update. The processor comprises: a control circuit for receiving and parsing instructions, and instructing a first arithmetic circuit to perform neural network training operations using FP12 format according to the parsed instructions; the first arithmetic circuit for performing forward operation, backward operation, and weight update in the neural network training using the FP12 format data, the first arithmetic circuit comprising a first exponent processing circuit and a first mantissa processing circuit, the first exponent processing circuit having a processing bit width of at least 8 bits, and the first mantissa processing circuit having a processing bit width of at least 3 bits; and a storage circuit for storing the weight update value obtained after the weight update, and using it as the weight for the next forward operation.

[0006] According to another aspect of the present invention, the FP12 format includes any one of the following formats: the FP12 format has a sign bit of 1 bit, an exponent bit of 8 bits, and a mantissa bit of 3 bits; the FP12 format has a sign bit of 1 bit, an exponent bit of 5 bits, and a mantissa bit of 6 bits; the FP12 format has a sign bit of 1 bit, an exponent bit of 4 bits, and a mantissa bit of 7 bits.

[0007] According to another aspect of the present invention, the processor further includes: a data format conversion circuit for converting the data to be processed in the training of the neural network into data in FP12 format.

[0008] According to another aspect of the present invention, in the provided processor, the data format conversion circuit is further configured to convert the result data of the inverse operation into high-precision data, the high-precision data having a higher precision than the FP12 format data; the processor further includes: a second arithmetic circuit configured to use the high-precision data to complete the weight update in the training of the neural network, the second arithmetic circuit including a second mantissa processing circuit, the processing bit width of the second mantissa processing circuit being at least 8 bits.

[0009] According to another aspect of the invention, the first mantissa processing circuit and the second mantissa processing circuit in the processor are multiplexed.

[0010] According to another aspect of the present invention, in the provided processor, the data format conversion circuit is further configured to convert the data to be processed in the nonlinear layer of the neural network into high-precision data, wherein the precision of the high-precision data is higher than that of the FP12 format data; the first operation circuit includes: a linear layer operation circuit, configured to perform the forward operation and the reverse operation of the linear layer in the neural network using the FP12 format data; the second operation circuit includes: a nonlinear layer operation circuit, configured to perform the forward operation and the reverse operation of the nonlinear layer in the neural network using the high-precision data.

[0011] According to another aspect of the present invention, the provided processor includes high-precision data in BF16 or FP32 format, wherein the BF16 format has a sign bit of 1, an exponent bit of 8, and a mantissa bit of 7, and the FP32 format has a sign bit of 1, an exponent bit of 8, and a mantissa bit of 23.

[0012] According to another aspect of the present invention, the provided processor further includes: a mixed-precision selection circuit, configured to receive and parse a first mixed-precision selection instruction, the first mixed-precision selection instruction being configured to instruct the execution of FP16 mixed-precision training or FP12 mixed-precision training; and a scaling factor setting circuit, configured to, when the first mixed-precision selection instruction instructs the data format conversion circuit to convert the data to be processed into the first FP12 format, and to instruct the first arithmetic circuit to set the scaling factor of the loss function in the inverse operation to 1 when performing the inverse operation using the first FP12 format, wherein the first FP12 format has a sign bit of 1, an exponent bit of 8, and a mantissa bit of 3.

[0013] According to another aspect of the present invention, in the provided processor, the mixed precision selection circuit is further configured to receive and parse a second mixed precision selection instruction, the second mixed precision selection instruction being used to instruct the execution of FP8 mixed precision training or FP12 mixed precision training; the processor further includes: a scaling factor setting circuit, configured to, when the second mixed precision selection instruction instructs the data format conversion circuit to quantize the data to be processed into the first FP12 format, set the scaling factor to 1, wherein the first FP12 format has 1 sign bit, 8 exponent bits, and 3 mantissa bits.

[0014] According to another aspect of the present invention, the provided processor includes: a linear layer forward operation circuit for performing forward operation of the linear layer in the training of the neural network using data in the first FP12 format; and a linear layer reverse operation circuit for performing reverse operation of the linear layer in the training of the neural network using data in the second FP12 format, wherein the second FP12 format has a sign bit of 1 bit, an exponent bit of 5 bits, and a mantissa bit of 6 bits.

[0015] The present invention also provides a processor for multi-machine, multi-card training, wherein each card in the multi-machine, multi-card training includes at least one processor as described in any of the above claims, and each processor uses its own scaling factor to complete the multi-machine, multi-card training.

[0016] According to another aspect of the present invention, a machine-readable medium is provided, wherein the API is executed by one or more processors, the API causing the one or more processors to perform neural network training, the neural network training comprising three stages: forward computation, backward computation, and weight update, the API causing the one or more processors to:

[0017] The processor receives and parses instructions, and instructs the first arithmetic circuit in the processor to perform neural network training operations using the FP12 format according to the parsed instructions.

[0018] The forward operation, backward operation, and weight update in the neural network training are completed on the first arithmetic circuit using the FP12 format data. The first arithmetic circuit includes a first exponent processing circuit and a first mantissa processing circuit. The processing bit width of the first exponent processing circuit is at least 8 bits, and the processing bit width of the first mantissa processing circuit is at least 3 bits.

[0019] Store the updated weight value obtained after the weight update, and use it as the weight for the next forward operation.

[0020] According to another aspect of the invention, the FP12 format includes any one of the following formats:

[0021] The FP12 format has a sign bit of 1, an exponent bit of 8, and a mantissa bit of 3.

[0022] The FP12 format has a sign bit of 1, an exponent bit of 5, and a mantissa bit of 6.

[0023] The FP12 format has a 1-bit sign bit, a 4-bit exponent bit, and a 7-bit mantissa bit.

[0024] According to another aspect of the invention, the API also enables the one or more processors to:

[0025] The data format conversion circuit in the processor is used to convert the data to be processed in the neural network training into FP12 format data.

[0026] According to another aspect of the invention, the API also enables the one or more processors to:

[0027] The data format conversion circuit is used to convert the result of the inverse operation into high-precision data, the high-precision data having a higher precision than the FP12 format data.

[0028] The high-precision data is used to update the weights in the neural network training on the second arithmetic circuit in the processing. The second arithmetic circuit includes a second mantissa processing circuit, and the processing bit width of the second mantissa processing circuit is at least 8 bits.

[0029] According to another aspect of the invention, the first mantissa processing circuit and the second mantissa processing circuit in the processor are multiplexed.

[0030] According to another aspect of the invention, the API also enables the one or more processors to:

[0031] The data format conversion circuit is used to convert the data to be processed in the nonlinear layer of the neural network into high-precision data, and the high-precision data has a higher precision than the FP12 format data.

[0032] Based on the linear layer operation circuit in the first operation circuit, the forward operation and the reverse operation of the linear layer in the neural network are completed using the FP12 format data;

[0033] Based on the nonlinear layer operation circuit in the second operation circuit, the forward operation and the reverse operation of the nonlinear layer in the neural network are completed using the high-precision data.

[0034] According to another aspect of the present invention, the high-precision data includes data in BF16 or FP32 format, wherein the BF16 format has 1 sign bit, 8 exponent bits, and 7 mantissa bits, and the FP32 format has 1 sign bit, 8 exponent bits, and 23 mantissa bits.

[0035] According to another aspect of the invention, the API also enables the one or more processors to:

[0036] The processor receives and parses a first mixed-precision selection instruction using a mixed-precision selection circuit, which is used to instruct whether to perform FP16 mixed-precision training or FP12 mixed-precision training.

[0037] When the first mixed precision selection instruction instructs to execute FP12 mixed precision training, the scaling sparsity setting circuit in the processor instructs the data format conversion circuit to convert the data to be processed into the first FP12 format, and instructs the first computing circuit to set the scaling factor of the loss function in the inverse operation to 1 when using the first FP12 format to complete the inverse operation. The first FP12 format has a sign bit of 1, an exponent bit of 8, and a mantissa bit of 3.

[0038] According to another aspect of the invention, the API also enables the one or more processors to:

[0039] The mixed precision selection circuit receives and parses a second mixed precision selection instruction, which is used to indicate whether to perform FP8 mixed precision training or FP12 mixed precision training.

[0040] When the second mixed precision selection instruction instructs the execution of FP12 mixed precision training, the scaling factor setting circuit in the processor is used to quantize the data to be processed into the first FP12 format and set the scaling factor to 1. The first FP12 format has 1 sign bit, 8 exponent bits, and 3 mantissa bits.

[0041] According to another aspect of the invention, the API also enables the one or more processors to:

[0042] Based on the linear layer forward operation circuit in the processor, the forward operation of the linear layer in the training of the neural network is completed using the data in the first FP12 format;

[0043] Based on the linear layer inversion operation circuit in the processor, the inversion operation of the linear layer in the training of the neural network is completed using data in the second FP12 format. The second FP12 format has a sign bit of 1 bit, an exponent bit of 5 bits, and a mantissa bit of 6 bits.

[0044] According to another aspect of the invention, each card in the multi-machine multi-card configuration includes at least one machine-readable medium as described in any of the preceding claims, on which an application programming interface (API) is stored, the API being executed by one or more processors, the API enabling each processor to perform the multi-machine multi-card training using its own scaling factor.

[0045] According to another aspect of the present invention, an artificial intelligence chip is also provided, the chip comprising the processor described in any one of the preceding claims.

[0046] According to another aspect of the present invention, an electronic device is also provided, the electronic device comprising the aforementioned artificial intelligence chip.

[0047] According to another aspect of the present invention, a system is also provided, comprising: a memory; one or more processors, wherein the memory stores an application programming interface (API) including any of the preceding claims.

[0048] According to another aspect of the invention, a method is also provided that responds to an application programming interface (API) including any of the foregoing claims.

[0049] The FP12 training method for neural networks provided in this invention offers advantages over FP32, BF16, and FP8 data formats in terms of both data representation range and accuracy. Using FP12 for neural network training reduces the storage space required at each level of the neural network during training, decreases computational latency, and improves processor efficiency while maintaining training accuracy. The circuitry for processing FP12 format can directly reuse existing circuitry for various data formats such as FP32 and BF16, reducing the design and manufacturing costs of the hardware for neural network training and improving the computational efficiency of the hardware.

[0050] Other features and aspects of the invention will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0051] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of the invention together with the specification and serve to explain the principles of the invention.

[0052] Figure 1 A structural diagram of a board according to an embodiment of the present invention is shown;

[0053] Figure 2 Show Figure 1 Structural diagram of the combined processing device in chip 101;

[0054] Figure 3 A structural diagram of a processor for performing neural network training according to an embodiment of the present invention is shown;

[0055] Figure 4 This diagram illustrates the FP12 data format provided in an embodiment of the present invention.

[0056] Figure 5 A structural diagram of a processor for performing neural network training according to another embodiment of the present invention is shown;

[0057] Figure 6 A structural diagram of a processor for performing neural network training according to another embodiment of the present invention is shown;

[0058] Figure 7 A schematic diagram showing the location of the application programming interface (API) according to another embodiment of the present invention is provided. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this invention are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" used in the specification and claims of this invention indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0061] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0062] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0063] Furthermore, to better illustrate the present invention, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that the present invention can be practiced without certain specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order to highlight the spirit of the invention.

[0064] In this embodiment of the invention, Figure 1 This is a structural diagram illustrating a board according to an embodiment of the present invention. In this embodiment, the processor or machine-readable medium for performing neural network training can be located within the board, specifically within a chip. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and substantial computing power.

[0065] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0066] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0067] Figure 2 Show Figure 1 A structural diagram of the combined processing device in chip 101. Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a DRAM 204.

[0068] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0069] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from on-chip storage device of computing device 201 and transmit it to processing device 203.

[0070] The processing device 203, as a general-purpose processing device, performs basic controls including but not limited to data transfer and starting / stopping the computing device 201. Depending on the implementation, the processing device 203 can be one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose and / or special-purpose processors. These processors include, but are not limited to, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure. The floating-point computing device provided by this invention can be disposed in the processing device 203.

[0071] DRAM 204 is used to store data to be processed. It is DDR memory, typically 16G or larger in size, and is used to store data in computing device 201 and / or processing device 203.

[0072] Figure 3 A structural diagram of a processor for performing neural network training according to an embodiment of the present invention is shown, such as... Figure 3 As shown, this invention provides a processor for performing neural network training, the neural network training comprising three stages: forward computation, backward computation, and weight update. The processor includes:

[0073] The control circuit 100 is used to receive and parse instructions, and instruct the first arithmetic circuit 200 to complete the neural network training operation using the FP12 format according to the parsed instructions.

[0074] The first arithmetic circuit 200 is used to complete the forward operation, backward operation and weight update in the training of the neural network using the FP12 format data. The first arithmetic circuit includes a first exponent processing circuit 210 and a first mantissa processing circuit 220. The processing bit width of the first exponent processing circuit is at least 8 bits, and the processing bit width of the first mantissa processing circuit is at least 3 bits.

[0075] The storage circuit 300 is used to store the updated weight value obtained after the weight update, and to use it as the weight for the next forward operation.

[0076] A neural network comprises at least one layer, each layer including multiple input nodes, multiple output nodes, and multiple connections connecting the input and output nodes. The connections between the input and output nodes are assigned numerical weights, each weight representing how the input to a given node relates to the output of that node. Each weight is multiplied by the input to a given node to generate the output. Various optimization methods, such as stochastic gradient descent, can be used to adjust the weights to alter the neural network's response to specific inputs.

[0077] In the forward pass phase of the neural network, the data to be processed (also called input data or training examples) representing the pre-classification dataset is sequentially passed through each layer of the neural network for computation. The data is propagated through the input nodes and weighted connections of each layer, and the output data is calculated through a series of data operations. In the backward pass phase, the error between the result data generated in the forward pass phase and the expected data is calculated using a loss function to obtain the result data of the backward pass. Commonly used loss functions include absolute value loss, logarithmic loss, mean squared error loss, and exponential loss. In the weight update phase, the weight data of the neural network is updated based on the result data of the backward pass, i.e., the error calculated using the loss function. The updated weight values ​​are stored and used as weights for the next forward pass. This process is repeated until the training result meets expectations, completing the training of the neural network. The trained neural network can be used for inference operations such as image classification and image recognition (in practical applications).

[0078] During the training and inference of neural networks, the amount of input data to be processed is large, and the data types are diverse. For the processor, the hardware carrier supporting neural network training and inference, the longer the bit width of the data types involved in various operations, the slower the processing speed. Therefore, the supported data types directly affect the processor's processing efficiency. Properly setting the processor's processing bit width can avoid idle and wasted hardware resources, minimizing the processor or chip area while ensuring processing efficiency, and maximizing the computing power provided by the processor or chip.

[0079] In this embodiment of the invention, the processor for performing neural network training includes a controller 100, a first arithmetic circuit 200, and a storage circuit 300.

[0080] The control circuit 100 coordinates and controls the operation of the first arithmetic circuit 200 and the storage circuit 300 to complete the training and inference process of the neural network. The control circuit 100 may include an instruction fetch circuit 110 and an instruction decoding circuit 120. The instruction fetch circuit 110 fetches ISA (Instruction Set Architecture) instructions, and the instruction decoding circuit 120 decodes the fetched instructions and sends the decoding result as control information to the first arithmetic circuit 200 and the storage circuit 300.

[0081] The first arithmetic circuit 200 includes a first exponent processing circuit 210 and a first mantissa processing circuit 220. The first exponent processing circuit 210 processes the exponent part of the floating-point data to be processed, and the first mantissa processing circuit 220 processes the mantissa part of the floating-point data to be processed.

[0082] The storage circuit 300 is used to store or transfer relevant data, including storing the data to be processed during the training and inference of the neural network, such as neuron and weight data, and also storing intermediate results generated during the training and inference of the neural network, including weight update values. In this embodiment of the invention, the storage circuit 300 is used to store the weight update values ​​obtained after weight updates, and uses them as weights for the next forward operation. In each iteration of the neural network training, the obtained weight update values ​​are stored in the storage circuit to complete the next iteration until the training of the neural network is complete.

[0083] In the training and inference of neural networks, the data to be processed is represented in two forms: fixed-point numbers and floating-point numbers.

[0084] Fixed-point numbers have a data format consisting of a 1-bit sign bit and multiple mantissa bits. The sign bit determines the sign of the fixed-point number. The mantissa bits determine the numerical value of the fixed-point number. A fixed-point number is a number in which the position of the decimal point is fixed. Fixed-point numbers are divided into fixed-point integers and fixed-point decimals. Since the position of the decimal point is fixed, it does not need to be represented; the value is calculated according to the agreed-upon position. Fixed-point numbers are usually represented as pure decimals or pure integers. If the value is a pure decimal, the decimal point is pre-positioned between the sign bit and the most significant mantissa bit. If the value is a pure integer, the decimal point is pre-positioned to the right of the least significant mantissa bit.

[0085] The format for representing floating-point numbers is standardized in IEEE 754. Taking a 37-bit single-precision floating-point number (FP37) as an example, it consists of 1 sign bit, 8 exponent bits (exp), and 28 mantissa bits, used to represent the following values:

[0086] Value = sign × mantissa × 2exp - 127

[0087] The exponent has 8 bits, which can represent the range from 0 to 255, making the exponent very large. Therefore, the IEEE 754 standard specifies an exponent offset of 127, shifting the exponent range to between -127 and 128, which is a more reasonable range. IEEE 754 further stipulates that there is an implicit bit to the left of the decimal point, which is usually 1. Therefore, the actual number of mantissa bits in the single-precision version is 29.

[0088] Because the above representation methods limit the range and precision of floating-point numbers, they can only be approximated for calculation, necessitating the consideration of rounding. In decimal, assuming you want to retain two decimal places (tenths and hundredths), the retained bit is the least significant bit (hundredths), the approximation bit is the first bit discarded (thousands), and all bits after the thousandths are called sticky bits; all information in the sticky bits is lost. For binary, if you want to retain two decimal places, the second bit to the right of the decimal point is the retained bit, the third bit to the right is the approximation bit, and all decimal places from the fourth bit to the right are sticky bits. Therefore, IEEE 754 defines four different rounding methods: rounding towards even numbers, rounding towards zero, rounding down, and rounding up. IEEE 754 defaults to rounding towards even numbers.

[0089] With the continuous development of neural network technology, the data formats for floating-point numbers have become increasingly diverse, including at least: FP32, TF32, BF16, FP16, UHP, and FP8. Among them, FP32 has 1 sign bit, 8 exponent bits, and 23 mantissa bits. TF32 has 1 sign bit, 8 exponent bits, and 10 mantissa bits. BF16 has 1 sign bit, 8 exponent bits, and 7 mantissa bits. FP16 has 1 sign bit, 5 exponent bits, and 10 mantissa bits. UHP has no sign bit, 6 exponent bits, and 10 mantissa bits. FP8 has two formats: one with 1 sign bit, 4 exponent bits, and 3 mantissa bits, and the other with 1 sign bit, 5 exponent bits, and 2 mantissa bits.

[0090] The width of the exponent bits in a floating-point number determines the data range, while the width of the mantissa bits determines the data precision. The data formats of floating-point numbers can be broadly divided into two categories: the first category has a longer mantissa bit width, such as FP32, TF32, BF16, FP16, and UHP; the second category has a shorter mantissa bit width, such as the two formats of FP8. Furthermore, the first type of floating-point number also has a longer exponent bit width, and the second type has a shorter exponent bit width. Different types of floating-point numbers, with different exponent and mantissa bit widths, different data ranges, and different data precisions, can adapt to various application scenarios in machine learning, the processing capabilities of various software or hardware platforms, or the complex requirements of different usage stages.

[0091] The first type of floating-point numbers guarantees both data range and precision, but their use in hardware and software environments requires significant resource overhead. In scenarios where precision requirements are not high, this leads to wasted computing, storage, and transmission resources, reduced computational efficiency, and a larger footprint on the chip or processor. The second type of floating-point numbers, while meeting precision requirements, improves the computational efficiency of hardware and software environments and reduces chip area, but suffers a significant loss in both data range and precision compared to the first type.

[0092] Figure 4 This diagram illustrates the FP12 data format provided in an embodiment of the present invention, as shown below. Figure 4 As shown, in this embodiment of the invention, a new data format FP12 is proposed. The data format of FP12 can be any one of the following three:

[0093] 1. Sign bit 1, exponent bits 8, mantissa bits 3. (This can be abbreviated as the first FP12 format or 183 format)

[0094] 2. The sign bit is 1 bit, the exponent is 5 bits, and the mantissa is 6 bits. (This can be simply referred to as the second FP12 format or 156 format)

[0095] 3. Sign bit 1, exponent bit 4, mantissa bit 7. (This can be abbreviated as the third FP12 format or 147 format)

[0096] Understandably, FP12 floating-point numbers in format 183 have a similar bit width for the mantissa as those in the second type of floating-point numbers mentioned above, but offer a larger data representation range. Similarly, FP12 floating-point numbers in formats 156 and 147 have a similar bit width for the exponent as those in the second type of floating-point numbers, but offer higher precision. Therefore, in addition to these two types of floating-point numbers, the FP12 format provides more flexible data representation for higher-level applications.

[0097] In this embodiment of the invention, the first arithmetic circuit 200 may include a first mantissa processing circuit with a processing width of at least 3 bits and an exponent processing circuit with a processing width of 8 bits. The mantissa processing circuit with a processing width of at least 3 bits is used to process the 3-bit mantissa bits in the first FP12 format, including various logical or mathematical operations such as multiplication, addition, shifting, and comparison. Similarly, the first arithmetic circuit 200 may also include a first mantissa processing circuit with a processing width of 6 bits, 7 bits, or 8 bits to process the mantissa bits in the second or third FP12 format. Likewise, a mantissa processing circuit with a width of 7 bits or more can be provided to process FP12 data in three different formats.

[0098] In neural network training and inference operations, vector multiplication and matrix multiplication / accumulation account for over 90% of the computations. Therefore, in the design of the processor's arithmetic circuits, the circuit design overhead of multiplication operations can be primarily considered. To support floating-point arithmetic, the arithmetic circuits within the processor typically include exponent processing circuits and mantissa processing circuits, used to handle various calculations of the exponent and mantissa bits of floating-point numbers, including logical and mathematical operations. The exponent processing circuits and mantissa processing circuits can include at least one more basic circuit structure such as adders, multipliers, shifters, comparators, etc., which this invention does not limit. To complete the multiplication operation, the exponent processing circuit is used to perform the addition of the exponent, and the mantissa processing circuit is used to perform the multiplication of the mantissa. Because the operation logic of the multiplication circuit is more complex than that of the addition and comparison circuits, the arithmetic unit and data storage require a larger area, resulting in greater hardware overhead. For multiplication circuits, longer-width multiplication circuits require more hardware overhead than shorter-width multiplication circuits. For example, the hardware overhead of a 10-bit wide multiplication circuit is much greater than that of a 3-bit wide multiplication circuit. Larger hardware overhead means that it requires more area or consumes more energy during operation, occupies more space when storing data, and requires more bandwidth when transmitting data.

[0099] In this embodiment of the invention, the first arithmetic circuit 200 utilizes FP12 format data to complete the three stages of forward operation, backward operation, and weight update in neural network training. For example, the entire neural network training process can be completed using FP12 data in 183 format. Due to the aforementioned advantages of the FP12 format, the first arithmetic circuit can utilize a mantissa processing circuit with a processing width of 3 bits or more to complete the operation on the FP12 format data to be operated on. Based on the advantages of the FP12 format in terms of data representation range and data precision, the processor including the first arithmetic circuit 200 can process data with a larger data representation range and has high computational efficiency. The FP12 format operation data can be operated on using a mantissa processing circuit with a width of at least 3 bits provided in this embodiment of the invention. A mantissa width of 3 bits is relatively short, the FP12 format data occupies less storage resources, and the processing efficiency of the mantissa processing circuit and the processor is also higher.

[0100] In this embodiment of the invention, the processor for performing neural network training includes a control circuit, a first arithmetic circuit, and a storage circuit. The processor uses the FP12 format data proposed in this invention to complete the neural network training process. Based on the characteristics of the FP12 format data itself, the data occupies less storage space during neural network training, and the processor's processing efficiency is also higher.

[0101] Figure 5 A structural diagram of a processor for performing neural network training according to another embodiment of the present invention is shown, such as Figure 5 As shown, the processor in this embodiment of the invention further includes a data format conversion circuit 400, used to convert the data to be processed in the neural network training into FP12 format data.

[0102] In this embodiment of the invention, the FP12 format data to be processed used by the processor when performing neural network training can be received directly by the processor. For example, the data to be processed in the neural network can be converted into FP12 format data on the software side and then sent to the processor. The processor can then directly perform the forward operation, backward operation and weight update stage of neural network training using the FP12 format data.

[0103] In this embodiment of the invention, the FP12 format data used by the processor when performing neural network training can also be that the software sends the data to be processed of the neural network model to the processor, the processor converts it into FP12 format data, and then performs the forward operation, backward operation, and weight update stage of neural network training.

[0104] In this embodiment of the invention, the processor may include a data format conversion circuit 400, which can be used to preprocess the input data to improve the computational efficiency of the neural network processor. Preprocessing may include data format conversion, normalization, or proportional scaling. Data format conversion converts the data into a floating-point or fixed-point number of any desired format. Normalization maps the input data to the range [0,1] or [-1,1] or a smaller range, such as [0.1,0.9]. Proportional scaling scales the input data according to the computational requirements. After the data format conversion circuit performs data format conversion, normalization, or proportional scaling, the computational efficiency of the input data can be improved. In this embodiment of the invention, the data format conversion circuit 400 can be used to convert the data to be processed in neural network training, including neuron data and weight data, from other formats to FP12 format, for example, to FP12 data in 183 format, and then transmit the converted FP12 format data to the first computation circuit 200 for computation to complete the neural network training. This invention does not limit the data format before conversion.

[0105] In this embodiment of the invention, the processor includes a data format conversion circuit that can convert received data of other formats to be processed into FP12 format data. Having the processor perform the data format conversion improves the processor's applicability.

[0106] Figure 6 A structural diagram of a processor for performing neural network training according to another embodiment of the present invention is shown, such as Figure 6 As shown, in the processor provided in this embodiment of the invention, the data format conversion circuit 400 is further used to convert the result data of the reverse operation into high-precision data, the precision of which is higher than that of the FP12 format data;

[0107] The processor further includes a second arithmetic circuit 500, used to update the weights in the training of the neural network using the high-precision data. The second arithmetic circuit includes a second mantissa processing circuit 520, and the processing bit width of the second mantissa processing circuit is at least 8 bits.

[0108] In neural network training, ensuring accurate weight updates is a crucial foundation. When training a neural network using lower precision, issues may arise such as: the weight update gradient obtained from the inverse computation being too small, causing underflow and resulting in a weight value of 0; the product of the learning rate and the weight update gradient being too small, also causing underflow and resulting in a weight update value of 0; or the weights being too large relative to their update values, leading to a weight update value of 0 due to insufficient precision of the lower precision data.

[0109] This invention proposes a mixed-precision training method for neural networks, which allows for weight update calculations using higher-precision data during the weight update phase of neural network training. For example, during training, the weights of each layer of the neural network can be saved in a higher-precision data format. During iterative calculations at each layer, forward and backward operations are performed according to the FP12 format. Then, during the weight update phase, the updated weight values ​​are calculated using the higher-precision data format and stored in a storage circuit.

[0110] In this embodiment of the invention, FP12 format data is used for computation during the forward and backward computation phases of the neural network, while higher precision data is used for computation during the weight update phase. For example, the high-precision data may include BF16 or FP32 format data. The BF16 format has 1 sign bit, 8 exponent bits, and 7 mantissa bits, while the FP32 format has 1 sign bit, 8 exponent bits, and 23 mantissa bits. Because higher precision data is used during the weight update phase, the accuracy of weight updates is improved, the number of iterations required for training or the total training time is shortened, and training efficiency is improved.

[0111] To handle higher-precision data operations, the processor in this embodiment further includes a second arithmetic circuit 500, which includes a second exponent processing circuit 510 and a second mantissa processing circuit 520. The second mantissa processing circuit has a processing width of at least 8 bits to support operations on BF16 or FP32 format data. For example, if BF16 format data is used for mixed-precision training, an 8-bit second mantissa processing circuit can be configured in the processor. If FP32 format data is used for mixed-precision training, a 24-bit second mantissa processing circuit can be configured. Alternatively, an 8-bit second mantissa processing circuit can be configured, and then the second mantissa processing circuit can be reused three times to complete one FP32 format data operation. This invention does not limit the scope of the invention.

[0112] The processor for performing mixed-precision training of neural networks proposed in this invention uses FP12 format data to perform forward and backward operations, and uses higher precision data to complete the weight update stage. The processor for performing neural network training has high processing efficiency, saves storage and computing resources, reduces the time required for neural network training, and improves the accuracy of training results.

[0113] In the processor of this embodiment, the first mantissa processing circuit and the second mantissa processing circuit are multiplexed.

[0114] In processor design, the configuration of the multiplication circuit directly affects various performance indicators such as processing efficiency and power consumption. In this embodiment of the invention, two arithmetic circuits are configured. The first arithmetic circuit processes low-precision floating-point numbers, with a shorter mantissa processing circuit. The second arithmetic circuit processes high-precision floating-point numbers, with a longer mantissa processing circuit. The first arithmetic circuit processes FP12 type floating-point numbers, and the second arithmetic circuit processes floating-point numbers with higher precision than FP12, including FP32, TF32, BF16, and FP16 types. The first and second arithmetic circuits work together to perform FP12 mixed-precision training.

[0115] To further save processor area and improve hardware utilization, the first and second arithmetic circuits can be reused. The mantissa processing circuit in the arithmetic circuit can be configured to process long-width mantissas. When processing short-width floating-point numbers, it can function as multiple short-width mantissa processing circuits simultaneously, doubling the processor's or chip's computing power through circuit reuse. Alternatively, the mantissa processing circuit can be configured to process short-width mantissas. When processing long-width floating-point numbers, a long-width mantissa is divided into multiple short-width mantissas, processed separately, and then combined to obtain the long-width floating-point result. For example, the mantissa processing circuit in the arithmetic circuit can be configured to process 8-bit mantissas. When processing FP12 data in 183 format, it can process two 183 data sets simultaneously. Alternatively, the mantissa processing circuit in the arithmetic circuit can be configured to process 3-bit mantissas. When processing BF16 data, the mantissa of the BF16 data can be calculated in three steps to obtain the final result. The multiplexing of the exponent processing circuit in the arithmetic circuit is the same as the multiplexing of the mantissa processing circuit described above, and will not be repeated here. The exponent processing circuit and the mantissa processing circuit in the arithmetic circuit can be configured to process long-width or short-width numbers separately, or they can be configured to process long-width or short-width numbers simultaneously, according to design requirements. This invention does not impose any limitations on this.

[0116] In the actual operation of a chip or processor, the on-chip and off-chip data transmission or data read / write bandwidth, as well as the data transmission or data read / write bandwidth between various on-chip storage media, can easily become bottlenecks in the chip's computing power. In this embodiment of the invention, while reusing the arithmetic circuit, the corresponding data transmission or data read bandwidth is configured according to the exponent and mantissa bit widths of the floating-point numbers to be supported by the arithmetic circuit, and corresponding data storage space is also configured to cooperate with the data operations of the arithmetic circuit and maximize the chip's computing power.

[0117] In this embodiment of the invention, the reuse of mantissa processing circuits and exponent processing circuits with different bit widths can reduce the design area of ​​the processor or chip and save hardware resource consumption.

[0118] In this embodiment of the invention, the data format conversion circuit is further configured to convert the data to be processed in the nonlinear layer of the neural network into high-precision data, wherein the precision of the high-precision data is higher than that of the FP12 format data.

[0119] The first computing circuit includes: a linear layer computing circuit, used to perform the forward and reverse operations of the linear layer in the neural network using the FP12 format data;

[0120] The second computing circuit includes a nonlinear layer computing circuit, used to perform the forward and reverse operations of the nonlinear layer in the neural network using the high-precision data.

[0121] Layers in a neural network can be divided into linear layers and nonlinear layers. Linear layers perform linear operations, while nonlinear layers perform nonlinear operations. Together, linear and nonlinear layers complete various complex operations in deep learning and obtain the desired results. For example, linear layers include fully connected layers, and nonlinear layers include activation layers. Due to the inherent algorithmic characteristics of nonlinear and linear operations, nonlinear layers require higher computational precision than linear layers to maintain the accuracy of the results.

[0122] In this embodiment of the invention, based on the first arithmetic circuit, a second arithmetic circuit is also provided to perform calculations on data with higher precision than FP12. The nonlinear layers in the neural network can utilize the second arithmetic circuit to perform forward and backward calculations during training, while the linear layers still utilize the first arithmetic circuit to perform forward and backward calculations during training. Furthermore, when the linear and nonlinear layers perform calculations with different precision using the first and second arithmetic circuits respectively, it can be based on the weight updates during neural network training using calculations with higher precision than FP12 (i.e., using the second arithmetic circuit to complete the weight updates during neural network training), or it can be based on the weight updates during neural network training using FP12 (i.e., using the first arithmetic circuit to complete the weight updates during neural network training). This invention does not limit the specific type of calculation. In this embodiment, data with higher precision than FP12 includes floating-point numbers of types such as FP32, TF32, BF16, and FP16, which are not limited to this type.

[0123] In this embodiment, the data format conversion circuit is used to perform mutual conversion between FP12 format and other precision formats according to the required precision of the data to be calculated when performing operations in the linear and nonlinear layers, so as to meet the calculation requirements of the linear and nonlinear layers.

[0124] In this embodiment, based on the operational characteristics of linear and nonlinear layers in a neural network, linear layers are operated on using FP12 format data, while nonlinear layers are operated on using higher-precision data, thus completing mixed-precision training of the neural network model. This improves computational efficiency and further enhances the accuracy of neural network training.

[0125] In this embodiment of the invention, the linear layer operation circuit includes:

[0126] A linear layer forward operation circuit is used to complete the forward operation of the linear layer in the training of the neural network using the data in the first FP12 format;

[0127] The linear layer inversion circuit is used to perform the inversion operation of the linear layer in the training of the neural network using data in the second FP12 format, wherein the sign bit of the second FP12 format is 1 bit, the exponent bit is 5 bits, and the mantissa bit is 6 bits.

[0128] As described above, based on the use of data with higher precision than FP12 to perform forward and backward operations in the nonlinear layers of the neural network, in this embodiment of the invention, the linear layer operation circuit includes a linear layer forward operation circuit and a linear layer backward operation circuit. Since the precision of the second FP12 format data is greater than that of the first FP12 format data, to further improve training accuracy during mixed-precision training of the neural network, the linear layer operations can be performed using the linear layer forward operation circuit and the linear layer backward operation circuit respectively in the forward and backward operation stages. Specifically, the linear layer forward operation circuit uses the first FP12 format data to perform the forward operation of the linear layer in the neural network training, while the linear layer backward operation circuit uses the second FP12 format data to perform the backward operation of the linear layer in the neural network training. The conversion between the first FP12 format, the second FP12 format, and other data formats is performed by a data format conversion circuit.

[0129] In this embodiment of the invention, by separating the forward and backward operations of the linear layer and using data in the second FP12 format with higher precision to complete the backward operation of the linear layer, the accuracy of the mixed precision training results of the neural network can be further improved.

[0130] In this embodiment of the invention, the processor further includes:

[0131] A mixed-precision selection circuit is used to receive and parse a first mixed-precision selection instruction, which is used to instruct whether to perform FP16 mixed-precision training or FP12 mixed-precision training.

[0132] The scaling factor setting circuit is used to instruct the data format conversion circuit to convert the data to be processed into the first FP12 format when the first mixed precision selection instruction instructs to execute FP12 mixed precision training, and to instruct the first operation circuit to set the scaling factor of the loss function in the inverse operation to 1 when using the first FP12 format to complete the inverse operation. The first FP12 format has a sign bit of 1, an exponent bit of 8, and a mantissa bit of 3.

[0133] In the embodiments of the present invention, it is understood that, based on the mutually reused or separately configured first and second operational circuits, the processor provided by the present invention can perform neural network mixed-precision training in different ways, such as FP12 mixed-precision training and FP16 mixed-precision training. The processor can determine which type of mixed-precision training needs to be performed by receiving and parsing the mixed-precision selection instruction. The instructions in this embodiment of the invention include ISA (Instruction Set Architecture) instructions. ISA instructions define the basic functions of the software and specify the functional goals for hardware implementation. They serve as the interface between software and hardware and are a crucial part of neural network system design. Users can include identifiers in the mixed-precision selection instructions, indicating which data type to use for mixed-precision training, based on the needs of the neural network. For example, the first mixed-precision selection instruction may include an identifier indicating whether to perform FP16 mixed-precision training or FP12 mixed-precision training. The processor, by parsing the identifiers in the instructions, converts the data to be processed into FP12 or higher precision data to perform mixed-precision training of the neural network, or converts the data to be processed into FP16 or higher precision data to perform mixed-precision training of the neural network. In different mixed-precision training methods, the data format conversion circuit performs a data quantization process when converting the data to be processed.

[0134] FP16 has half the bit width of FP32. Compared to using FP32 data for neural network training, the memory occupied by weights and other parameters in neural network operations is also half, saving a significant amount of memory space and improving processor efficiency. However, FP16 mixed-precision training also brings problems such as precision overflow and rounding errors. Because the effective data representation range of FP16 is much smaller than that of FP32, overflow and underflow can occur when using FP16 to replace high-precision data. In neural network training, it is necessary to calculate the gradient of the weights. Underflow can occur when the gradient is smaller than the weight value, preventing the neural network from converging. To solve underflow, FP16 introduces a loss scale. The loss value obtained from the forward calculation is amplified during the backward calculation. That is, after multiplying the parameters by the loss scale, the decimal places that might overflow are shifted forward to the range that FP16 can represent, thus avoiding underflow.

[0135] Because the effective data representation range of the FP12 proposed in this invention is very large, especially when using data in the first FP12 format (1 sign bit, 8 exponent bits, and 3 mantissa bits), underflow is extremely rare, and null values ​​(NaN) are not easily encountered during training. Therefore, FP12 mixed-precision training does not require loss scaling. Thus, when the mixed-precision selection circuit receives a first mixed-precision selection instruction instructing the execution of FP16 mixed-precision training, the scaling factor setting circuit needs to set the corresponding loss scaling loss factor to match the FP16 mixed-precision training. When the mixed-precision selection circuit receives a first mixed-precision selection instruction instructing the execution of FP12 mixed-precision training, the scaling factor of the loss function in the inverse operation is set to 1.

[0136] In this embodiment of the invention, the processor can execute various mixed-precision training methods. The mixed-precision selection circuit receives and parses a first mixed-precision selection instruction, which instructs the processor to execute either FP16 mixed-precision training or FP12 mixed-precision training. When the first mixed-precision selection instruction instructs the processor to execute FP12 mixed-precision training, the scaling factor setting circuit instructs the data format conversion circuit to convert the data to be processed into the first FP12 format and set the scaling factor of the loss function in the inverse operation to 1. The FP12 data format proposed in this invention can reduce the computational load during mixed-precision training, improve the processor's computational efficiency, and simultaneously improve the accuracy of mixed-precision training of neural networks.

[0137] In the processor of this embodiment of the invention, the mixed precision selection circuit is further configured to receive and parse a second mixed precision selection instruction, the second mixed precision selection instruction being configured to instruct the execution of FP8 mixed precision training or the execution of FP12 mixed precision training;

[0138] The processor also includes:

[0139] The scaling factor setting circuit is used to instruct the data format conversion circuit to quantize the data to be processed into the first FP12 format when the second mixed precision selection instruction instructs to execute FP12 mixed precision training, and to set the scaling factor to 1, wherein the first FP12 format has 1 sign bit, 8 exponent bits, and 3 mantissa bits.

[0140] In the embodiments of the present invention, it is understood that, based on the mutually reused or separately configured first and second operational circuits, the processor provided by the present invention can also perform mixed-precision training of neural networks in different modes, such as FP8 mixed-precision training. The processor can determine the mixed-precision training to be performed by receiving and parsing a mixed-precision selection instruction. For example, the second mixed-precision selection instruction may carry an identifier indicating whether to perform FP8 mixed-precision training or FP12 mixed-precision training. The processor, by parsing the identifier in the instruction, converts the data to be processed into FP12 or higher precision data to perform mixed-precision training of the neural network, or converts the data to be processed into FP8 or higher precision data to perform mixed-precision training of the neural network. In different modes of mixed-precision training, the data format conversion circuit performs a data quantization process when converting the data to be processed.

[0141] FP8 has two formats: one format has 1 sign bit, 4 exponent bits, and 3 mantissa bits, abbreviated as E4M3; the other has 1 sign bit, 5 exponent bits, and 2 mantissa bits, abbreviated as E5M2. For some higher-level applications, the data range represented by FP8 is insufficient for data representation. Furthermore, to reduce quantization errors in FP8 data, a scaling factor needs to be set when quantizing the data to be processed into FP8. In FP8 mixed-precision training of neural networks, all vectors involving FP8 forward and reverse operations need to be quantized into FP8 data using a scaling factor, which will not be elaborated further in this invention. One method for calculating the scaling factor is to use the maximum value from the past N iterations, or the maximum value from the most recent iteration. Therefore, calculating the scaling factor involves complex algorithms, a large amount of data, and requires additional storage space to store historical data information.

[0142] Because the FP12 proposed in this invention has a large effective data representation range, especially when using data in the first FP12 format (1 sign bit, 8 exponent bits, and 3 mantissa bits), it avoids the problems that occur when quantizing to the FP8 data format. Therefore, the scaling factor can be set to 1 for FP12 mixed-precision training. When the mixed-precision selection circuit receives a first mixed-precision selection instruction instructing the execution of FP8 mixed-precision training, the scaling factor setting circuit needs to set the corresponding scaling factor to match the FP8 quantization. When the mixed-precision selection circuit receives a first mixed-precision selection instruction instructing the execution of FP12 mixed-precision training, the scaling factor can be set to 1.

[0143] In this embodiment of the invention, the processor can execute various mixed-precision training methods. The mixed-precision selection circuit receives and parses a second mixed-precision selection instruction, which instructs the processor to execute either FP8 mixed-precision training or FP12 mixed-precision training. When the first mixed-precision selection instruction instructs the processor to execute FP12 mixed-precision training, the scaling factor setting circuit instructs the data format conversion circuit to quantize the data to be processed into the first FP12 format and sets the scaling factor to 1. The FP12 data format proposed in this invention can reduce the computational load during mixed-precision training, improve the processor's computational efficiency, save data storage space, and simultaneously improve the accuracy of mixed-precision training of neural networks.

[0144] This invention also provides a processor for multi-machine, multi-card training, wherein each card in the multi-machine, multi-card training includes at least one processor as described above, and each processor uses its own scaling factor to complete the multi-machine, multi-card training.

[0145] With the development of neural network technology, neural network models are becoming increasingly large, and the volume of data to be processed is also extremely large. To accelerate model training, the training task can be split and distributed across multiple computing nodes using certain methods. Then, the information to be summarized can be aggregated using a different method, thereby accelerating the training process. Common splitting methods include data splitting and model splitting. The split data or model is distributed across multiple processors for parallel processing, which can significantly improve the training speed. Examples include single-machine multi-GPU training and multi-machine multi-GPU training methods used in practical engineering. A computing node (also called a machine) represents a physical node, such as a computer, server, or computing device. Each computing node includes multiple processors (called GPUs). Multi-machine multi-GPU training methods include the DDP (Distributed Processing Module) mode, where each GPU corresponds to a separate neural network model (i.e., a process). In multi-machine, multi-GPU training, data splitting involves dividing a batch of data across different machines or GPUs. This also includes splitting or replicating a neural network model (a process) across multiple computing nodes for parallel computation, and grouping processes for distributed training. Regardless of the parallelism method, multi-machine, multi-GPU systems require addressing the issue of data consistency. Therefore, frequent data transfers between multiple machines and GPUs severely impact the efficiency of neural network training.

[0146] In this embodiment of the invention, based on the characteristics of the proposed FP12 format, when the processor performs FP12 mixed precision training, the scaling factor in the FP12 process can be set to 1. Different processors (cards) can use the local scaling factor within a group or between different computing nodes, avoiding the synchronous transmission of data generated by scaling factor-related operations, saving a lot of data storage space, and improving the training efficiency of multi-machine and multi-card systems.

[0147] Figure 7 The diagram illustrates the location of an application programming interface (API) according to another embodiment of the present invention. In one embodiment, a machine-readable medium is also provided storing the API, which is executed by one or more processors. The API causes the one or more processors to perform neural network training, the neural network training comprising three stages: forward computation, backward computation, and weight update. The API causes the one or more processors to:

[0148] The processor receives and parses instructions, and instructs the first arithmetic circuit in the processor to perform neural network training operations using the FP12 format according to the parsed instructions.

[0149] The forward operation, backward operation, and weight update in the neural network training are completed on the first arithmetic circuit using the FP12 format data. The first arithmetic circuit includes a first exponent processing circuit and a first mantissa processing circuit. The processing bit width of the first exponent processing circuit is at least 8 bits, and the processing bit width of the first mantissa processing circuit is at least 3 bits.

[0150] Store the updated weight value obtained after the weight update, and use it as the weight for the next forward operation.

[0151] An API (Application Programming Interface) is a software application interface. An API can be distributed or otherwise provided as part of one or more libraries, runtimes, drivers, or any other software or executable code sets. An API can also be a set of software instructions provided by a higher-level software framework, or an interface to a neural network model supported by a higher-level software framework. In other words, an API can be an interface possessed by various layers of software running on top of a hardware platform; this invention does not limit this. When executed, an API can cause one or more processors to perform various operations. Based on user-implemented software programs, one or more APIs can be used to perform various operations, such as device management and mathematical operations. Users can perform various operations on the processor by calling APIs to complete higher-level applications.

[0152] API interfaces can send instructions to processors, and instruct the processors and various circuits within the processor to perform various operations predefined in the API. In one embodiment of the invention, a machine-readable medium is provided storing an API, which is executed by one or more processors. The API causes the one or more processors to perform neural network training, which includes three stages: forward computation, backward computation, and weight update.

[0153] In embodiments of the present invention, the API instructs the processor to perform various operations, which are the same as those described above in the processor provided by the present invention, and will not be repeated here.

[0154] In one embodiment of the present invention, the FP12 format includes any one of the following formats:

[0155] The FP12 format has a sign bit of 1, an exponent bit of 8, and a mantissa bit of 3.

[0156] The FP12 format has a sign bit of 1, an exponent bit of 5, and a mantissa bit of 6.

[0157] The FP12 format has a 1-bit sign bit, a 4-bit exponent bit, and a 7-bit mantissa bit.

[0158] In one embodiment of the present invention, the API further enables the one or more processors to convert the data to be processed in the neural network training into FP12 format data using a data format conversion circuit in the processor.

[0159] In one embodiment of the invention, the API also enables the one or more processors to:

[0160] The data format conversion circuit is used to convert the result of the inverse operation into high-precision data, the high-precision data having a higher precision than the FP12 format data.

[0161] The high-precision data is used to update the weights in the neural network training on the second arithmetic circuit in the processing. The second arithmetic circuit includes a second mantissa processing circuit, and the processing bit width of the second mantissa processing circuit is at least 8 bits.

[0162] In one embodiment of the present invention, the first mantissa processing circuit and the second mantissa processing circuit in the processor are multiplexed.

[0163] In one embodiment of the invention, the API also enables the one or more processors to:

[0164] The data format conversion circuit is used to convert the data to be processed in the nonlinear layer of the neural network into high-precision data, and the high-precision data has a higher precision than the FP12 format data.

[0165] Based on the linear layer operation circuit in the first operation circuit, the forward operation and the reverse operation of the linear layer in the neural network are completed using the FP12 format data;

[0166] Based on the nonlinear layer operation circuit in the second operation circuit, the forward operation and the reverse operation of the nonlinear layer in the neural network are completed using the high-precision data.

[0167] In one embodiment of the present invention, the high-precision data includes data in BF16 or FP32 format, wherein the BF16 format has 1 sign bit, 8 exponent bits, and 7 mantissa bits, and the FP32 format has 1 sign bit, 8 exponent bits, and 23 mantissa bits.

[0168] In one embodiment of the invention, the API also enables the one or more processors to:

[0169] The processor receives and parses a first mixed-precision selection instruction using a mixed-precision selection circuit, which is used to instruct whether to perform FP16 mixed-precision training or FP12 mixed-precision training.

[0170] When the first mixed precision selection instruction instructs to execute FP12 mixed precision training, the scaling sparsity setting circuit in the processor instructs the data format conversion circuit to convert the data to be processed into the first FP12 format, and instructs the first computing circuit to set the scaling factor of the loss function in the inverse operation to 1 when using the first FP12 format to complete the inverse operation. The first FP12 format has a sign bit of 1, an exponent bit of 8, and a mantissa bit of 3.

[0171] In one embodiment of the invention, the API also enables the one or more processors to:

[0172] The mixed precision selection circuit receives and parses a second mixed precision selection instruction, which is used to indicate whether to perform FP8 mixed precision training or FP12 mixed precision training.

[0173] When the second mixed precision selection instruction instructs the execution of FP12 mixed precision training, the scaling factor setting circuit in the processor is used to quantize the data to be processed into the first FP12 format and set the scaling factor to 1. The first FP12 format has 1 sign bit, 8 exponent bits, and 3 mantissa bits.

[0174] In one embodiment of the invention, the API also enables the one or more processors to:

[0175] Based on the linear layer forward operation circuit in the processor, the forward operation of the linear layer in the training of the neural network is completed using the data in the first FP12 format;

[0176] Based on the linear layer inversion operation circuit in the processor, the inversion operation of the linear layer in the training of the neural network is completed using data in the second FP12 format. The second FP12 format has a sign bit of 1 bit, an exponent bit of 5 bits, and a mantissa bit of 6 bits.

[0177] In one embodiment of the present invention, each card in the multi-machine multi-card configuration includes at least one machine-readable medium as described in any one of claims 12-21, on which an application programming interface (API) is stored, the API being executed by one or more processors, the API enabling each processor to perform the multi-machine multi-card training using its own scaling factor.

[0178] In one embodiment of the present invention, an artificial intelligence chip is also provided, the chip comprising the processor described in any one of the foregoing claims.

[0179] In one embodiment of the present invention, an electronic device is also provided, the electronic device including the artificial intelligence chip described above.

[0180] In one embodiment of the present invention, a system is also provided, comprising:

[0181] Memory;

[0182] One or more processors,

[0183] The memory stores application programming interface (API) as described in any of the above-mentioned items.

[0184] In one embodiment of the invention, a method is also provided that responds to an application programming interface (API) including any of the foregoing claims.

[0185] The electronic devices in the embodiments of the present invention include data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, mobile phones, dashcams, navigators, sensors, cameras, servers, cloud servers, cameras, camcorders, projectors, watches, headphones, mobile storage, wearable devices, vehicles, home appliances, and / or medical devices.

[0186] The means of transportation include airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical equipment includes MRI scanners, ultrasound scanners and / or electrocardiographs.

[0187] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A processor for performing neural network training, the processor comprising: The neural network training includes three stages: forward computation, backward computation, and weight update. The processor includes: The control circuit is used to receive and parse instructions, and instruct the first arithmetic circuit to complete the neural network training operation using the FP12 format according to the parsed instructions; The first arithmetic circuit is used to perform forward and reverse operations in the training of the neural network using the FP12 format data. The first arithmetic circuit includes a first exponent processing circuit and a first mantissa processing circuit. The processing bit width of the first exponent processing circuit is at least 8 bits, and the processing bit width of the first mantissa processing circuit is at least 3 bits. The storage circuit stores the updated weight values ​​obtained after the weight update, and uses them as the weights for the next forward calculation. The processor also includes: A data format conversion circuit is used to convert the data to be processed in the neural network training into FP12 format data; The data format conversion circuit is also used to convert the result data of the reverse operation into high-precision data, the high-precision data having a higher precision than the FP12 format data. The processor also includes: The second arithmetic circuit is used to update the weights in the training of the neural network using the high-precision data. The second arithmetic circuit includes a second mantissa processing circuit, and the processing bit width of the second mantissa processing circuit is at least 8 bits.

2. The processor according to claim 1, characterized in that, The FP12 format includes any one of the following formats: The FP12 format has a sign bit of 1, an exponent bit of 8, and a mantissa bit of 3. The FP12 format has a sign bit of 1, an exponent bit of 5, and a mantissa bit of 6. The FP12 format has a 1-bit sign bit, a 4-bit exponent bit, and a 7-bit mantissa bit.

3. The processor according to claim 1, characterized in that, The first mantissa processing circuit and the second mantissa processing circuit in the processor are multiplexed.

4. The processor according to claim 1, characterized in that, The data format conversion circuit is also used to convert the data to be processed in the nonlinear layer of the neural network into high-precision data, wherein the precision of the high-precision data is higher than that of the FP12 format data. The first operational circuit includes: A linear layer operation circuit is used to perform the forward and reverse operations of the linear layer in the neural network using the FP12 format data. The second operational circuit includes: A nonlinear layer operation circuit is used to perform the forward and reverse operations of the nonlinear layer in the neural network using the high-precision data.

5. The processor according to claim 1, characterized in that, The high-precision data includes data in BF16 or FP32 format. The BF16 format has 1 sign bit, 8 exponent bits, and 7 mantissa bits. The FP32 format has 1 sign bit, 8 exponent bits, and 23 mantissa bits.

6. The processor according to claim 1, characterized in that, The processor also includes: A mixed-precision selection circuit is used to receive and parse a first mixed-precision selection instruction, which is used to instruct whether to perform FP16 mixed-precision training or FP12 mixed-precision training. The scaling factor setting circuit is used to instruct the data format conversion circuit to convert the data to be processed into the first FP12 format when the first mixed precision selection instruction instructs to execute FP12 mixed precision training, and to instruct the first operation circuit to set the scaling factor of the loss function in the inverse operation to 1 when using the first FP12 format to complete the inverse operation. The first FP12 format has a sign bit of 1, an exponent bit of 8, and a mantissa bit of 3.

7. The processor according to claim 6, characterized in that, The mixed precision selection circuit is also used to receive and parse a second mixed precision selection instruction, which is used to indicate whether to perform FP8 mixed precision training or FP12 mixed precision training. The processor also includes: The scaling factor setting circuit is used to instruct the data format conversion circuit to quantize the data to be processed into the first FP12 format when the second mixed precision selection instruction instructs to execute FP12 mixed precision training, and to set the scaling factor to 1, wherein the first FP12 format has 1 sign bit, 8 exponent bits, and 3 mantissa bits.

8. The processor according to claim 4, characterized in that, The linear layer computing circuit includes: A linear layer forward operation circuit is used to complete the forward operation of the linear layer in the training of the neural network using data in the first FP12 format, wherein the first FP12 format has 1 sign bit, 8 exponent bits, and 3 mantissa bits; The linear layer inversion circuit is used to perform the inversion operation of the linear layer in the training of the neural network using data in the second FP12 format, wherein the sign bit of the second FP12 format is 1 bit, the exponent bit is 5 bits, and the mantissa bit is 6 bits.

9. A processor for multi-machine, multi-GPU training, characterized in that, Each card in the multi-machine multi-card configuration includes at least one processor as described in any one of claims 1-8, and each processor uses its own scaling factor to complete the multi-machine multi-card training.

10. A machine-readable medium having an application programming interface (API) stored thereon, characterized in that, The API is executed by one or more processors, which instruct the processors to perform neural network training. The neural network training includes three stages: forward computation, backward computation, and weight update. The API enables the processors to: The processor receives and parses instructions, and instructs the first arithmetic circuit in the processor to perform neural network training operations using the FP12 format according to the parsed instructions. The forward and reverse operations in the training of the neural network are completed on the first arithmetic circuit using the FP12 format data. The first arithmetic circuit includes a first exponent processing circuit and a first mantissa processing circuit. The processing bit width of the first exponent processing circuit is at least 8 bits, and the processing bit width of the first mantissa processing circuit is at least 3 bits. Store the updated weight value obtained after the weight update, and use it as the weight for the next forward calculation. The API also enables the one or more processors: The data format conversion circuit in the processor is used to convert the data to be processed in the neural network training into FP12 format data. The API also enables the one or more processors: The data format conversion circuit is used to convert the result of the inverse operation into high-precision data, the high-precision data having a higher precision than the FP12 format data. The high-precision data is used to update the weights in the neural network training on the second arithmetic circuit in the processor. The second arithmetic circuit includes a second mantissa processing circuit, and the processing bit width of the second mantissa processing circuit is at least 8 bits.

11. The machine-readable medium according to claim 10, characterized in that, The FP12 format includes any one of the following formats: The FP12 format has a sign bit of 1, an exponent bit of 8, and a mantissa bit of 3. The FP12 format has a sign bit of 1, an exponent bit of 5, and a mantissa bit of 6. The FP12 format has a 1-bit sign bit, a 4-bit exponent bit, and a 7-bit mantissa bit.

12. The machine-readable medium according to claim 10, characterized in that, The first mantissa processing circuit and the second mantissa processing circuit in the processor are multiplexed.

13. The machine-readable medium according to claim 10, characterized in that, The API also enables the one or more processors: The data format conversion circuit is used to convert the data to be processed in the nonlinear layer of the neural network into high-precision data, and the high-precision data has a higher precision than the FP12 format data. Based on the linear layer operation circuit in the first operation circuit, the forward operation and the reverse operation of the linear layer in the neural network are completed using the FP12 format data; Based on the nonlinear layer operation circuit in the second operation circuit, the forward operation and the reverse operation of the nonlinear layer in the neural network are completed using the high-precision data.

14. The machine-readable medium according to claim 10, characterized in that, The high-precision data includes data in BF16 or FP32 format. The BF16 format has 1 sign bit, 8 exponent bits, and 7 mantissa bits. The FP32 format has 1 sign bit, 8 exponent bits, and 23 mantissa bits.

15. The machine-readable medium according to claim 10, characterized in that, The API also enables the one or more processors: The processor receives and parses a first mixed-precision selection instruction using a mixed-precision selection circuit, which is used to instruct whether to perform FP16 mixed-precision training or FP12 mixed-precision training. When the first mixed precision selection instruction instructs the execution of FP12 mixed precision training, the scaling sparsity setting circuit in the processor instructs the data format conversion circuit to convert the data to be processed into the first FP12 format, and instructs the first computing circuit to set the scaling factor of the loss function in the inverse operation to 1 when using the first FP12 format to complete the inverse operation. The first FP12 format has a sign bit of 1, an exponent bit of 8, and a mantissa bit of 3.

16. The machine-readable medium according to claim 15, characterized in that, The API also enables the one or more processors: The mixed precision selection circuit receives and parses a second mixed precision selection instruction, which is used to indicate whether to perform FP8 mixed precision training or FP12 mixed precision training. When the second mixed precision selection instruction instructs the execution of FP12 mixed precision training, the scaling factor setting circuit in the processor is used to quantize the data to be processed into the first FP12 format and set the scaling factor to 1. The first FP12 format has 1 sign bit, 8 exponent bits, and 3 mantissa bits.

17. The machine-readable medium according to claim 15, characterized in that, The API also enables the one or more processors: Based on the linear layer forward computation circuit in the processor, the forward computation of the linear layer in the neural network training is completed using the data in the first FP12 format; Based on the linear layer inversion operation circuit in the processor, the inversion operation of the linear layer in the training of the neural network is completed using data in the second FP12 format. The second FP12 format has a sign bit of 1 bit, an exponent bit of 5 bits, and a mantissa bit of 6 bits.

18. A machine-readable medium for multi-machine, multi-card training, characterized in that, Each card in the multi-machine, multi-card configuration includes at least one machine-readable medium as described in any one of claims 10-17, on which an application programming interface (API) is stored, the API being executed by one or more processors, the API enabling each processor to perform the multi-machine, multi-card training using its own scaling factor.

19. An artificial intelligence chip, characterized in that, The chip includes a processor as described in any one of claims 1-9.

20. An electronic device, characterized in that, The electronic device includes the artificial intelligence chip as described in claim 19.

21. A system comprising: Memory; One or more processors, The memory stores the application programming interface (API) as described in any one of claims 10-18.

22. A method in response to an application programming interface (API) including any one of claims 10-18.

Images