A processing device, apparatus, method and related products thereof

Abstract

The present disclosure discloses a system on chip for neural network operation, a device, a method and related products, including a computer program product. The system on chip for neural network operation can be applied to a computing processing device included in a combined processing device, which can include one or more data processing devices. The aforementioned combined processing device can also include an interface device and other processing devices. The computing processing device interacts with other processing devices to jointly complete a user-specified computing operation. The combined processing device can also include a storage device connected to the device and other processing devices, respectively, for storing data of the device and other processing devices. By converting the data type of the operation result of the neural network, the algorithm accuracy is improved, and the computing power consumption and cost are reduced. In addition, the present disclosure also improves the performance and accuracy of the intelligent computing system as a whole.

Description

Technical Field

[0001] This disclosure generally relates to the field of artificial intelligence. More specifically, this disclosure relates to a processing apparatus, device, method for neural network operations, and related products. Background Technology

[0002] Supporting one or more specific data types is a fundamental and crucial function of computing systems. From a hardware perspective, supporting a data type requires designing various units on the hardware, such as arithmetic processing units and decoding control units, specifically for that data type. This design undoubtedly increases the circuit area of ​​the hardware, resulting in higher power consumption. From a software perspective, supporting a data type necessitates modifications to the underlying compiler, function libraries, and the top-level software stack. For intelligent computing systems, the use of different data types can also affect the accuracy of algorithms. Therefore, the choice of data type has a significant impact on the hardware design, software stack, and algorithm accuracy of intelligent computing systems. Given this, improving the algorithm accuracy of intelligent computing systems with relatively low hardware efficiency and limited software stack support is a pressing technical challenge. Summary of the Invention

[0003] In view of the technical problems mentioned in the background section above, this disclosure provides a processing apparatus, device, method for neural network operation, and related products in several aspects. Specifically, the solution of this disclosure improves algorithm accuracy and reduces computational power consumption and cost by converting the data type of the neural network operation result into a preset data type with lower data precision, suitable for data storage and transfer within the system-on-chip and / or between the system-on-chip and off-chip systems, under conditions of lower hardware area and power consumption and software stack support. In addition, the solution disclosed in this disclosure also improves the overall performance and accuracy of the intelligent computing system. The neural network of the embodiments of this disclosure can be applied to various fields, such as image processing, speech processing, text processing, etc., and these processes may include, but are not limited to, recognition and classification.

[0004] In a first aspect, this disclosure provides a processing apparatus, comprising: an arithmetic unit configured to perform at least one arithmetic operation to obtain an arithmetic result; and a first type converter configured to convert the data type of the arithmetic result into a third data type; wherein the data precision of the data type of the arithmetic result is greater than the data precision of the third data type, and the third data type is suitable for storing and transporting the arithmetic result.

[0005] In a second aspect, this disclosure provides an edge device for neural network operations, including the system-on-chip of the first aspect of this disclosure, and configured to participate in performing training and / or inference operations of the neural network at the edge device.

[0006] In a third aspect, this disclosure provides a cloud device for neural network operations, including a system-on-a-chip of the first aspect of this disclosure, and configured to participate in performing training and / or inference operations of a neural network at the cloud device.

[0007] In a fourth aspect, this disclosure provides a cloud-edge collaborative computing neural network system, comprising: a cloud computing subsystem configured to perform neural network-related operations in the cloud; an edge computing subsystem configured to perform neural network-related operations at the edge; and a processing device according to a first aspect of this disclosure, wherein the processing device is disposed at the cloud computing subsystem and / or the edge computing subsystem, and configured to participate in the execution of the training process of the neural network and / or the inference process based on the neural network.

[0008] In a fifth aspect, this disclosure provides a method for neural network computation, implemented by a processing device, and comprising: performing at least one computation operation to obtain a computation result; converting the data type of the computation result into a third data type; wherein the data precision of the data type of the computation result is greater than the data precision of the third data type, and the third data type is suitable for storing and transporting the computation result.

[0009] In a sixth aspect, this disclosure provides a computer program product including a computer program that, when executed by a processor, implements a system-on-a-chip based on the first aspect of this disclosure.

[0010] By utilizing the processing apparatus, devices, methods for neural network computation, and related products provided above, the solution disclosed herein improves algorithm accuracy and reduces computational power consumption and cost by converting the data type of the neural network computation results into a preset data type with lower data precision, suitable for data storage and transfer within the system-on-chip and / or between the system-on-chip and off-chip systems, thereby achieving this with extremely low hardware area and power consumption costs and software stack support. Furthermore, the solution disclosed herein also improves the overall performance and accuracy of the intelligent computing system. Attached Figure Description

[0011] The above and other objects, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the present disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0012] Figure 1 A schematic diagram illustrating an example of the convolution operation process is shown;

[0013] Figure 2 A schematic diagram illustrating an example of the max pooling operation process is shown;

[0014] Figure 3 A schematic diagram illustrating an example of a fully connected operation process is shown;

[0015] Figure 4 A functional block diagram of a processing apparatus according to an embodiment of the present disclosure is shown;

[0016] Figure 5 A schematic diagram of an example of a 32-bit floating-point number is shown;

[0017] Figure 6 A schematic diagram showing an example of a 16-bit floating-point number is provided.

[0018] Figure 7 A functional block diagram of a processing apparatus according to another embodiment of the present disclosure is shown;

[0019] Figure 8 This diagram illustrates the internal structure of the processing device of this disclosure when it is a multi-core architecture;

[0020] Figure 9 A schematic diagram of an example of a TF32 floating-point number is shown;

[0021] Figure 10 A flowchart illustrating a method for neural network computation according to an exemplary embodiment of this disclosure is shown;

[0022] Figure 11 A structural diagram of a combined processing apparatus according to an embodiment of this disclosure is shown; and

[0023] Figure 12 A schematic diagram of the structure of a board according to an embodiment of this disclosure is shown. Detailed Implementation

[0024] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0025] Artificial Neural Networks (ANNs), also simply called Neural Networks (NNs), are mathematical models of algorithms that mimic the behavioral characteristics of animal neural networks to perform distributed parallel information processing. A neural network is a machine learning algorithm that includes at least one neural network layer. The types of neural network layers include convolutional layers, fully connected layers, pooling layers, activation layers, batch normalization (BN) layers, and so on. A brief description of the various layers relevant to the scheme disclosed herein is provided below.

[0026] The convolutional layers of a neural network can perform convolution operations, which can be the inner product of the input feature matrix and the convolution kernel. Figure 1 A schematic diagram illustrating an example of the convolution operation process is shown. For example... Figure 1 As shown, the input to the convolutional layer is a feature matrix X, which is 6×6 in size; the convolutional kernel K is a 3×3 matrix. To calculate the first value Y of the output matrix Y... 0,0 By placing the center of the convolution kernel K at position (1, 1) of matrix X, and multiplying the coefficients of matrix X at the corresponding position with the coefficients of the convolution kernel one by one, and then summing them, we can obtain the following formula and result:

[0027] Y 0,0 =X 0,0 ×K 0,0 +X 0,1 ×K 0,1 +X 0,2 ×K 0,2 +X 1,0 ×K 1,0 +X 1,1 ×K 1,1 +X 1,2 ×K 1,2 +X 2,0 ×K 2,0 +X 2,1 ×K 2,1 +X 2,2 ×K 2,2 =2×2+3×3+1×2+2×2+3×3+1×2+2×2+3×3+1×2=45.

[0028] Pooling layers in neural networks perform pooling operations to reduce the number of parameters and computational cost, thus suppressing overfitting. Pooling operators include max pooling, average pooling, and L1 pooling. 2 Pooling, etc. For ease of understanding... Figure 2 A schematic diagram illustrating an example of the max pooling operation process is shown. For example... Figure 2As shown, the pooling window is 3×3 with a stride of 3. The maximum value of 5 is found in the 3×3 submatrix at the top left corner of the input feature map as the first output. The pooling window is then moved 3 grids to the right on the input feature map to find the maximum value of 5 as the second output. All output values ​​can be obtained by continuing to slide the pooling window down.

[0029] Fully connected layers in a neural network can perform fully connected operations. Fully connected operations map high-dimensional features into a one-dimensional feature vector, which contains all the feature information of the high-dimensional features. Similarly, for ease of understanding, Figure 3 A schematic diagram illustrating an example of a fully connected operation process is shown. For example... Figure 3 As shown, the input to the fully connected layer is a feature matrix X, which is 3×3 in size. To calculate the first value Y of the output matrix Y... 0,0 To obtain the following formula, we need to multiply all the coefficients of matrix X and the weight corresponding to each position, and then sum them up.

[0030] Y 0,0 =X 0,0 ×W 0,0 +X 0,1 ×W 0,1 +X 0,2 ×W 0,2 +X 1,0 ×W 1,0 +X 1,1 ×W 1,1 +X 1,2 ×W 1,2 +X 2,0 ×W 2,0 +X 2,1 ×W 2,1 +X 2,2 ×W 2,2 .

[0031] The activation layers of a neural network can perform activation operations, which can be implemented using activation functions. Activation functions include the sigmoid function, tanh function, ReLU function, PReLU function, ELU function, and so on. Activation functions can provide non-linear characteristics to the neural network.

[0032] The Batch Normalization (BN) layer in a neural network can perform batch normalization (BN) operations. By normalizing the input using multiple samples, the input can be normalized to a standard normal distribution with added parameters. The batch normalization process is as follows:

[0033] If the input of a certain neural network layer is x i (i = 1, ..., M, where M is the size of the training set), x i =[x i1 ;xi2 ;...;x id Let x be a d-dimensional vector. First, consider x... i Normalize each dimension k:

[0034]

[0035] Then, the normalized values ​​are scaled and offset to obtain the data after BN transformation:

[0036]

[0037] Where, γ k and β k Scaling and offset parameters for each dimension.

[0038] It should be noted that this disclosure is for illustrative purposes only, using convolutional layers, fully connected layers, pooling layers, activation layers, and batch normalization (BN) layers of neural networks to illustrate the computational operations of neural networks. This disclosure is not limited to the aforementioned types of neural network operations in any situation. Specifically, computational operations involving other types of neural network layers (such as Long Short-Term Memory (“LSTM”) layers, Local Response Normalization (“LRN”) layers, etc.) fall within the scope of this disclosure.

[0039] The specific embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0040] Figure 4 A functional block diagram of a processing apparatus according to an embodiment of the present disclosure is shown. Figure 4 As shown, the processing device 400 includes an arithmetic unit 401, a first type converter 402, a memory 403, and a controller 404. In one implementation scenario, the controller 404 can be used to control the coordinated operation of the arithmetic unit 401 and the memory 403 to complete the machine learning task. The arithmetic unit 401 can be used to perform at least one arithmetic operation and obtain the result. Optionally, the arithmetic unit can be used to perform neural network-related arithmetic operations, including but not limited to multiplication, addition, activation, and other arithmetic operations. The result obtained by the arithmetic unit can be the result obtained by the arithmetic unit performing a portion of the arithmetic operations. Alternatively, the result obtained by the arithmetic unit can also be the result obtained by the arithmetic unit performing all the arithmetic operations. The memory 403 can be used to store or transfer data. According to the disclosed solution, the first type converter 402 can be used to convert the data type of the result obtained by the arithmetic unit 401 into the data type of the result. The data precision of the data type of the result obtained by the arithmetic unit 401 can be greater than the data precision of the third data type, and the third data type is suitable for storing and transferring the above-mentioned result.

[0041] The data in neural networks includes various data types, such as integers, floating-point numbers, complex numbers, Booleans, strings, and quantized integers. These data types can be further subdivided according to their precision (i.e., bit length in the context of this disclosure). For example, integer data includes 8-bit integers, 16-bit integers, 32-bit integers, and 64-bit integers; floating-point data includes half-precision (float16) floating-point numbers, single-precision (float32) floating-point numbers, and double-precision (float64) floating-point numbers; complex number data includes 64-bit single-precision complex numbers and 126-bit double-precision complex numbers; and quantized integer data includes quantized 8-bit integers (qint8), quantized 16-bit integers (qint16), and quantized 32-bit integers (qint32).

[0042] To facilitate understanding of the meaning of data precision in this disclosure Figure 5 A schematic diagram of an example of a 32-bit floating-point number is shown. For example... Figure 5 As shown, a 32-bit floating-point number (single precision) consists of 1 sign bit (s), 8 exponent bits (e), and 23 mantissa bits (m). The sign bit s = 0 represents a positive sign, and the sign bit s = 1 represents a negative sign. The exponent bit e ranges from 0 to 255, and the mantissa m is also called the fractional part. Figure 5 The true value of the number represented in binary is "(-1)(1.1001000011111101)×2 128-127 ", converted to decimal is "-3.132720947265625".

[0043] Figure 6 Another illustration shows an example of a 16-bit floating-point number. For example... Figure 6 As shown, a 16-bit floating-point number (half precision) consists of 1 sign bit (s), 5 exponent bits (e), and 10 mantissa bits (m). The sign bit s = 0 represents a positive sign, and the sign bit s = 1 represents a negative sign. The exponent bit e ranges from 0 to 31, and the mantissa m is also called the fractional part. Figure 6 The true value of the number represented in binary is "(-1)(1.1001)×2 16-15 ", converted to decimal is "-3.125".

[0044] For floating-point data types, the precision is related to the number of bits in the mantissa (m); the more bits in the mantissa (m), the higher the precision. Therefore, it can be understood that the precision of a 32-bit floating-point number is greater than that of a 16-bit floating-point number. Considering this, the arithmetic unit 401 disclosed herein can use a higher-precision data type, such as a 32-bit single-precision floating-point number, when performing neural network operations. After obtaining the higher-precision calculation result, the arithmetic unit 401 can transmit the result to the first type converter 402, which performs the conversion from high-precision data to low-precision data.

[0045] Although neural networks use high data precision during computation to ensure algorithm accuracy in practical applications, higher precision requires greater bandwidth and storage space. Therefore, in this disclosed solution, memory 403 uses low-bit-width, low-precision data types to store or move data. Correspondingly, the third data type can be one of these low-bit-width or low-precision data types used for data storage or movement in memory 403, such as the TF32 floating-point number described in detail below. Based on the foregoing considerations, in this embodiment, the first type converter 402 can perform the conversion from high-precision computation results to low-precision third data types. It should be clear that the low bit-width and low precision of the data type here are relative to the bit-width and precision of the data type used by the arithmetic unit to perform computational operations.

[0046] Figure 7 A functional block diagram of a processing apparatus for neural network operations according to another embodiment of the present disclosure is shown. Based on the foregoing description, those skilled in the art will understand... Figure 7 The processing device shown can be Figure 4 One possible specific implementation of the processing device shown, therefore previously combined Figure 4 The description of the processing device also applies to the following text in conjunction with... Figure 7 The description.

[0047] like Figure 7As shown, the processing device 700 includes an arithmetic unit 401, a first type converter 402, a memory 403, and a controller 404. The arithmetic unit 401 includes a first arithmetic unit 4011 and a second arithmetic unit 4012. The first arithmetic unit 4011 is configured to perform a first type operation on a first data type to obtain the result of the first type operation. The second arithmetic unit 4012 is configured to perform a second type operation on the result of the first type operation using a second data type to obtain the result of the second type operation, and to perform nonlinear layer operations on the result of the second type operation to obtain the result of the nonlinear layer operation on the second data type. The first arithmetic unit 4011 and the second arithmetic unit 4012 can be vector operators or matrix operators; no specific limitation is made here.

[0048] When data and operations in a neural network are represented using a data type with a certain precision, the computing units in the hardware need to adapt to this precision, for example, they can use an arithmetic unit with that precision. In this embodiment, the first data type has a first precision, the second data type has a second precision, and the third data type has a third precision. The first arithmetic unit 4011 can be a first precision arithmetic unit, and the second arithmetic unit 4012 can be a second precision arithmetic unit. For example, the first arithmetic unit 4011 can be a 16-bit floating-point arithmetic unit, and the second arithmetic unit 4012 can be a 32-bit floating-point arithmetic unit. Here, the first type of operation can be a certain type of operation in the neural network (e.g., pooling operation), or it can be a specific type of operation (e.g., multiplication operation); the second type of operation can be a certain type of operation in the neural network (e.g., convolution operation), or it can be a specific type of operation (e.g., addition operation). Optionally, the first type of operation can be multiplication operation, and the second type of operation can be addition operation. In this case, the first data precision can be less than the second data precision, and the third data precision can be less than the first data precision and / or the second data precision.

[0049] In this embodiment, the first type converter 402 is further configured to convert the result of the nonlinear layer operation into the result of the operation of a third data type. As an example, the aforementioned result of the nonlinear layer operation may have a second data precision, and the second data precision may be greater than the third data precision.

[0050] In other embodiments, the first data type has a low bit length data precision, the second data type has a high bit length data precision, and the third data type has a data precision less than and / or less than the data precision of the first data type and / or the second data type. Optionally, the third data type has a data precision between the low bit length of the first data type and the high bit length of the second data type. In the context of this disclosure, the bit length of a data type refers to the number of bits required to represent that data type. Taking a 32-bit floating-point number as an example, representing a 32-bit floating-point number requires 32 bits, therefore the bit length of a 32-bit floating-point number is 32. Similarly, the bit length of a 16-bit floating-point number is 16. Based on this, the bit length of the second data type is higher than that of the first data type, and the bit length of the third data type is higher than that of the first data type and lower than that of the second data type. Optionally, the first data type may include a 16-bit floating-point number with a bit length of 16 bits, the second data type may include a 32-bit floating-point number with a bit length of 32 bits, and the third data type may include a TF32 floating-point number with a bit length of 19 bits.

[0051] To facilitate understanding of the data precision of TF32 floating-point numbers in this disclosure Figure 9 A schematic diagram of an example of a TF32 floating-point number is shown. For example... Figure 9 As shown, a TF32 floating-point number consists of 1 sign bit (s), 8 exponent bits (e), and 10 mantissa bits (m). The sign bit s = 0 represents a positive sign, and the sign bit s = 1 represents a negative sign. The exponent bit e ranges from 0 to 255, and the mantissa m is also called the fractional part. Figure 9 The true value of the number represented in binary is "(-1)(1.1001)×2 128-127 ", converted to decimal is "-3.125". TF32 floating-point numbers use the same 10-bit mantissa as 16-bit floating-point numbers and the same 8-bit exponent as 32-bit floating-point numbers. Because TF32 floating-point numbers use the same 10-bit mantissa as 16-bit floating-point numbers, they can meet the precision requirements of neural network algorithms; and because TF32 floating-point numbers also use the same 8-bit exponent as 32-bit floating-point numbers, they can support the same number range as 32-bit floating-point numbers.

[0052] As another embodiment of the third data type, it may also include the truncated half-precision floating-point number bf16. bf16 consists of 1 sign bit (s), 8 exponent bits (e), and 7 mantissa bits (m). The meanings of the sign bit, exponent bit, and mantissa bit of bf16 are the same as or similar to those of 16-bit floating-point numbers and 32-bit floating-point numbers, and therefore will not be described again here.

[0053] When the third data type is bf16, the second arithmetic unit 4012 can perform a second-type operation on the result of the first-type operation using TF32 floating-point numbers to obtain the result of the second-type operation. Then, non-linear layer operations of the neural network can be performed on the result of the second-type operation to obtain the non-linear layer operation result of TF32 floating-point numbers. Subsequently, depending on the scenario or requirements of the operation, the first type converter 402 can further convert the non-linear layer operation result of TF32 floating-point numbers into a bf16 non-linear layer operation result.

[0054] It should be noted that the memory 403 in this disclosure can use TF32 floating-point numbers or bf16 to store or transfer data. Furthermore, by converting the result of a nonlinear layer operation with a second data precision into the aforementioned TF32 floating-point operation result, the solution disclosed in this disclosure can reduce the power consumption and cost of computation, and also improve the overall performance and accuracy of the intelligent computing system.

[0055] In other embodiments, the first type converter 402 is also configured for data type conversion between different operations of the neural network. Since different operations of the neural network may use data types with different data precisions (e.g., convolution operations use 16-bit floating-point data types, activation operations use 32-bit floating-point data types), the first type converter 402 can be used for data type conversion between operations with different data precisions. This data type conversion can be either from a high-precision operation to a low-precision operation, or from a low-precision operation to a high-precision operation.

[0056] In other embodiments, the first type converter 402 is further configured to convert the result of a third data type operation into a first data type or a second data type to facilitate subsequent operations by the first or second arithmetic unit. Specifically, the first type converter 402 can convert the result of a neural network operation performed by the arithmetic unit 401 into a third data type operation result and store it in the memory 403. If the controller 404 issues an instruction to continue performing neural network operations on the result of the third data type operation, the memory 403 can send the result of the third data type operation to the first type converter 402 to perform data type conversion, and send the obtained first or second data type operation result to the arithmetic unit 401 to perform subsequent neural network operation operations. If the first type converter 402 converts the result of the third data type operation into a first data type operation result, the first arithmetic unit 4011 can perform subsequent neural network operation operations; if the first type converter 402 converts the result of the third data type operation into a second data type operation result, the second arithmetic unit 4012 can perform subsequent neural network operation operations.

[0057] In some embodiments, the processing device 700 further includes a second type converter 405, configured to convert the operation result of a third data type into a first data type or a second data type, so as to facilitate subsequent operations by the first or second arithmetic unit. The first type converter 402 can convert the operation result obtained by the arithmetic unit 401 performing neural network operations into the operation result of a third data type and store it in the memory 403. If the controller 404 issues an instruction to continue performing neural network operations on the operation result of the third data type, the memory 403 can send the operation result of the third data type to the second type converter 405 to perform data type conversion, and send the obtained operation result of the first or second data type to the arithmetic unit 401 to perform subsequent neural network operations. If the second type converter 405 converts the operation result of the third data type into the operation result of the first data type, the first arithmetic unit 4011 can perform subsequent neural network operations; if the second type converter 405 converts the operation result of the third data type into the operation result of the second data type, the second arithmetic unit 4012 can perform subsequent neural network operations.

[0058] In some embodiments, the first type converter 402 and / or the second type converter 405 are configured to truncate the operation result according to the nearest neighbor truncation method or a preset truncation method to achieve data type conversion. The nearest neighbor truncation method is explained below using a decimal number as an example. If the third data type is a floating-point number 3.4, and the first or second data type is an integer, the data conversion process of the first type converter 402 is as follows: find the nearest integer 3 (3) to the floating-point number 3.4, and convert the floating-point number 3.4 into the integer 3. If the third data type is the integer 3, and the first or second data type is a floating-point number with a decimal precision, the data conversion process of the second type converter 405 is as follows: find the nearest floating-point number 3.1 or 2.9 to the integer 3, and convert the integer 3 into 3.1 or 2.9.

[0059] Depending on the implementation scenario, the preset truncation method can be any user-configured truncation method. The following uses a decimal number as an example to illustrate one preset truncation method. Assume the third data type disclosed herein is a floating-point number 3.5, and the first or second data type is an integer, with the preset truncation method being to find the nearest integer upwards. Based on this assumed scenario, the data conversion process of the first type converter 402 disclosed herein could be: finding the nearest integer upwards to the floating-point number 3.5, i.e., the integer 4, and then converting the floating-point number 3.5 into the integer 4. Similarly, if the third data type is the integer 3, and the first or second data type is a floating-point number with a decimal precision, then the data conversion process of the second type converter 405 could be: finding the nearest floating-point number upwards to the integer 3, such as the floating-point number 3.1, and then converting the integer 3 into 3.1.

[0060] As can be seen from the above description, the first type converter 402 and / or the second type converter 405 disclosed herein can perform data type conversion based on either the nearest neighbor truncation method or a preset truncation method. Additionally or alternatively, the first type converter 402 and / or the second type converter 405 can also perform data type conversion based on a combination of the nearest neighbor truncation method and a preset truncation method. Therefore, this disclosure does not limit the type or usage of the truncation method.

[0061] In other embodiments, the processing device 700 further includes at least one on-chip memory 4031, wherein the on-chip memory may be internal memory of the processing device. Depending on the implementation, the processing device 700 of this disclosure may be implemented as a single-core processor or a processor with a multi-core architecture.

[0062] Figure 8A schematic diagram of the internal structure of the processing device 700 when it is a multi-core processor architecture is shown. For ease of description, the processing device 700 with a multi-core architecture will be referred to as the multi-core processing device 800 below. According to the scheme of this disclosure, the computing resources of the multi-core processing device 800 can be designed in a hierarchical structure and can be implemented as a system-on-a-chip. Furthermore, it can include at least one cluster, and each cluster can include multiple processor cores. Each processor core can include at least one computing module 824-m, and each computing module can be at least one of the above-mentioned multipliers, adders, nonlinear arithmetic units, etc.

[0063] like Figure 8 As shown, the storage resources of the multi-core processing device 800 can also adopt a hierarchical structure design. Each processing core 811 may have a local storage module 823 required for performing computing tasks. This local storage module 823 may specifically include NRAM and WRAM (not shown in the figure). Each cluster 85 may have a shared storage module, which can be accessed by multiple processor cores 811-n within the cluster. Specifically, the local storage module 823 within each processing core can interact with the shared storage module via a communication module 821. Multiple clusters can be connected to one or more off-chip DRAMs 808, allowing the shared storage module within each cluster to interact with the DRAMs 808, and the processor cores within each cluster to interact with the off-chip DRAMs 808 via a communication module 822.

[0064] In one embodiment, a processor core in a multi-core processing device 800 can be used for at least one computation operation to obtain a computation result. This computation result can be converted into a third data type and transferred and stored in the form of the third data type among the various levels of storage resources of the multi-core processing device 800. Specifically, the third data type of this disclosure (such as TF32) moves the computation result from the local storage module to SRAM and temporarily stores it in SRAM. When subsequent computations of the processor core require the computation result (i.e., there is a dependency between the preceding and following computations), the temporarily stored third data type, such as TF32, can be converted into a first or second data type required by the processor core for computation. Alternatively, if it is determined that subsequent computations of the processor core require the computation result, the computation result can be temporarily stored in the local storage module or SRAM in its original data type (first or second data type), thereby reducing data conversion operations.

[0065] Due to limited on-chip storage space, when the operation result will not be reused, it can also be stored on off-chip DRAM. In one scenario, the operation result is temporarily stored in the local storage module or SRAM in its original data type (first or second data type). In this case, when the operation result will not be reused, it can be converted to a third data type and stored in the off-chip DRAM. In another scenario, the processing core converts the obtained operation result into third data type data after completing the relevant operation. In this case, when the operation result will not be reused, the third data type operation result stored in the local storage module or SRAM can be stored in the off-chip DRAM. Optionally, to further reduce I / O overhead during the process of storing data in off-chip DRAM, the operation result of the third data type can be compressed.

[0066] Depending on the specific computational scenario, the various devices disclosed herein can be used individually or in combination to perform various computations. For example, the processing apparatus of this disclosure can be applied to forward inference and backward training computations of neural networks. Specifically, in some embodiments, one or more of the first arithmetic unit 4011, the second arithmetic unit 4012, the first type converter 402, and the second type converter 405 of this disclosure are configured to perform one or more of the following operations: computations on the output neurons of the neural network inference process; gradient propagation computations on the neural network training process; and weight update computations on the neural network training process. For ease of understanding, the training, forward and backward propagation, and update operations of the neural network are briefly described below.

[0067] Training a neural network involves adjusting the parameters of its hidden and output layers to make the calculated results approximate the true results. The training process primarily includes two stages: forward propagation and backward propagation. In forward propagation (also known as forward inference), the input target is used to calculate hidden layers using weights, biases, and activation functions. These hidden layers then use the weights, biases, and activation functions of the next layer to obtain the next hidden layer. Through this iterative process, the input feature vector is progressively extracted from low-level features to abstract features, ultimately outputting the target classification result. The basic principle of backward propagation is to first calculate the loss function based on the forward propagation results and the true values. Then, using gradient descent and the chain rule, the partial derivatives of the loss function with respect to each weight and bias are calculated, representing the impact of the weights or biases on the loss. Finally, the weights and biases are updated. Here, the process of calculating on the output neurons based on the trained neural network model is the operation of the output neurons in the neural network inference process. Backpropagation in the neural network training process includes gradient propagation operations and weight update operations.

[0068] In some embodiments, during the aforementioned neural network inference and / or training processes, the first type of operation disclosed herein may include multiplication, the second type of operation may include addition, and the nonlinear layer operation may include activation. The multiplication operation here can be either a multiplication operation within a convolution operation or a multiplication operation within a fully connected operation. Similarly, the addition operation here can be either an addition operation within a convolution operation or an addition operation within a fully connected operation. This disclosure does not limit the type of neural network operation to multiplication or addition. Furthermore, the aforementioned nonlinear layer may be an activation layer of the neural network.

[0069] Similar to the specific operations described above, during the neural network inference process and / or neural network training process, the first arithmetic unit 4011 of this disclosure can perform a first type operation on a first data type to obtain the result of the first type operation. Correspondingly, the second arithmetic unit 4012 performs a second type operation on the result of the first type operation using a second data type to obtain the result of the second type operation. The second arithmetic unit then performs nonlinear layer operations on the neural network based on the result of the second type operation, thereby obtaining the nonlinear layer operation result of the second data type. As mentioned earlier, the first data type can have a first data precision, while the second data type can have a second data precision, and the first data precision is less than the second data precision. Subsequently, the first type converter 402 converts the nonlinear layer operation result into the operation result of a third data type. Here, the data precision of the third data type can be less than the first data precision or the second data precision.

[0070] For example, a neural network may include convolutional layers and activation layers. During the forward inference operation of the neural network, the arithmetic unit can first perform convolutional layer operations (including multiplication and addition operations) to obtain the convolution operation result. A first type converter can convert the data type of the convolution operation result to a third data type so that the operation result can be stored in on-chip memory or moved to off-chip memory. For example, the data type of the input data for the convolutional layer operation is FP16, and the data type of the convolution operation result is TF32. Next, the arithmetic unit of the processing device can use the convolution operation result as input to perform activation layer operations. At this time, a first type converter or a second type converter can convert the convolution operation result of the third data type to the data type required by the arithmetic unit of the processing device to perform the activation layer operation. For example, a first data type converter or a second type converter is used to convert the convolution operation result of data type TF32 to the data type FP16 or FP32 required by the activation layer operation. The arithmetic unit can then perform activation layer operations based on the convolution operation result to obtain the activation layer operation result. The first type converter can convert the data type of the activation layer operation result to a third data type, so that the activation layer operation result can be stored in on-chip memory or moved to off-chip memory. For example, the first data type converter is used to convert the data type of the activation layer operation result from FP32 to TF32.

[0071] In one embodiment, since there is a data dependency between convolutional layer operations and activation layer operations, the intermediate results of each operation can be stored in on-chip storage to reduce I / O overhead. In this case, the data type conversion process for intermediate results such as convolutional operation results can be omitted, thereby reducing the number of on-chip data conversions and improving computational efficiency.

[0072] Furthermore, the processing device can calculate a loss function based on the activation operation results. During the backward propagation of the neural network, the processing device can calculate the output gradient of the activation layer based on the loss function, and then perform gradient propagation and weight update operations based on the output gradient. In the gradient propagation operation, the processor of the processing device can calculate the gradient of the input layer of the current output layer based on the output gradient of the current output layer and the weight data. The gradient of each input layer can be used as a calculation result. A first-type converter can convert the data type of the calculation result to a third-type data type to store the calculation result in on-chip storage or move it to off-chip storage. Of course, when there are data dependencies between the operations of different layers of the neural network, the intermediate results of each operation can also be stored in on-chip storage to reduce I / O overhead. In this case, the data type conversion process of intermediate results such as the gradient of the convolutional layer can be omitted, thereby reducing the number of on-chip data conversions and improving computational efficiency.

[0073] In the weight update operation, the processing device can calculate the inter-layer weight update gradient based on the output gradient of the current output layer and the neurons in the input layer of the current output layer. Each inter-layer weight update gradient can be considered a computation result. A first-type converter can convert the data type of this result to a third-type data type to store the result in on-chip memory or move it to off-chip memory. Then, the processing device can calculate the updated weight data based on the weight update gradient and the weight data before the update (which can be stored in off-chip memory as a third-type data type). At this point, the first-type or second-type converter of the processing device can convert the weight update gradient (a third-type data type) and the weight data before the update into the data type required by the processing device's arithmetic unit to perform the weight update. The arithmetic unit can then perform the operation based on the weight update gradient and the updated weight data to obtain the updated weights. Finally, the first-type converter can convert the data type of the updated weights back to a third-type data type to store the updated weights in off-chip memory.

[0074] In some embodiments, a 16-bit floating-point arithmetic unit (equivalent to the first arithmetic unit of this disclosure) can be used to perform multiplication operations (i.e., the first type of operation in this disclosure) in neural network operations, and then a 32-bit floating-point arithmetic unit (equivalent to the second arithmetic unit of this disclosure) can be used to perform addition operations on the results of the multiplication operations (i.e., the second type of operation in this disclosure). After the aforementioned multiplication and addition operations are completed, a 32-bit floating-point convolution result is output. Next, a 32-bit floating-point arithmetic unit is used to perform nonlinear layer operations on the convolution result in the activation layer of the neural network model. For the obtained 32-bit floating-point nonlinear layer operation result, the 32-bit floating-point nonlinear layer operation result can be converted into a TF32 floating-point number (i.e., the third data type in this disclosure) nonlinear layer operation result according to the nearest neighbor principle and a user-configurable truncation method.

[0075] In some scenarios, the on-chip system can perform data transfer between off-chip memory (such as DRAM) and on-chip memory (SRAM), between on-chip memory (SRAM) and between off-chip memory (such as DRAM). In other scenarios, when the neural network model still needs to perform operations on the results of non-linear layer operations of TF32 floating-point numbers, the results of non-linear layer operations of TF32 floating-point numbers can be converted into 16-bit floating-point non-linear layer operation results and / or 32-bit floating-point non-linear layer operation results according to the nearest neighbor principle and / or a user-configurable truncation method.

[0076] In some embodiments, the processing apparatus 400 of this disclosure may further include a compressor configured to compress the calculation results at a third data precision for data storage and transfer within the on-chip system and / or between the on-chip system and an off-chip system. In one scenario, the compressor may be located between the arithmetic unit 403 and the memory 401 for performing data type conversion (e.g., conversion for a third data type) to facilitate data storage and transfer within the on-chip system and / or between the on-chip system and an off-chip system.

[0077] Depending on the application scenario, the system-on-a-chip (SoC) of this disclosure can be flexibly deployed at appropriate locations within an artificial intelligence system, such as the edge layer and / or the cloud. In view of this, this disclosure also provides an edge device for neural network computation, comprising the SoC according to any one of the exemplary embodiments of this disclosure, and configured to participate in the execution of neural network training and / or inference operations at the edge device. The edge device here may include devices such as cameras, smartphones, gateways, wearable computing devices, and sensors at the network edge. Similarly, this disclosure also provides a cloud device for neural network computation, comprising the SoC according to any one of the exemplary embodiments of this disclosure, and configured to participate in the execution of neural network training and / or inference operations at the cloud device. The cloud device here includes cloud servers or boards implemented based on cloud technology. Here, the aforementioned cloud technology can refer to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network (WAN) or local area network (LAN) to realize data computation, storage, processing, and sharing.

[0078] In addition, this disclosure also provides a cloud-edge collaborative computing neural network system, including: a cloud computing subsystem configured to perform neural network-related operations in the cloud; an edge computing subsystem configured to perform neural network-related operations at the edge; and a system-on-a-chip according to any one of the exemplary embodiments of this disclosure, wherein the system-on-a-chip is disposed at the cloud computing subsystem and / or the edge computing subsystem and configured to participate in the training process of the neural network and / or the inference process based on the neural network.

[0079] After introducing the on-chip system with exemplary embodiments of this disclosure, the following references are made to... Figure 10 A method for neural network computation according to an exemplary embodiment of this disclosure is described.

[0080] like Figure 10As shown, the method 1000 for neural network computation is implemented by an on-chip system. In step S1001, at least one computation operation is performed to obtain the computation result. In step S1002, the data type of the computation result is converted into the computation result of a third data type. The data precision of the data type of the computation result is greater than that of the third data type. The third data type is suitable for data storage and transfer within the on-chip system and / or between the on-chip system and an off-chip system.

[0081] Given the steps of method 1000 and Figure 4 The operation of the intermediate processing unit 400 is the same, therefore regarding Figure 4 The description of the processing device 400 also applies to the operation of method 1000, and therefore further details regarding the operation of method 1000 will not be repeated here. Furthermore, although other steps of method 1000 are not described herein for the sake of brevity, those skilled in the art will understand from the foregoing description that method 1000 may also include... Figure 7 or Figure 8 The diagram shows the various steps of the operation performed by the on-chip system.

[0082] Figure 11 This is a structural diagram illustrating a combined processing apparatus according to an embodiment of the present disclosure. It will be understood that the combined processing apparatus disclosed herein can be used to perform the data type conversion operations described for the foregoing of this disclosure in conjunction with the accompanying drawings. In some scenarios, the combined processing apparatus may include the system-on-a-chip described for the foregoing of this disclosure in conjunction with the accompanying drawings. In other scenarios, the combined processing apparatus may be connected to the system-on-a-chip described for the foregoing of this disclosure in conjunction with the accompanying drawings to execute an executable program obtained according to the method for neural network operations described above.

[0083] like Figure 11 As shown, the combined processing device 1100 includes a computing processing device 1102, an interface device 1104, other processing devices 1106, and a storage device 1108. Depending on the application scenario, the computing processing device may include one or more computing devices 1110, which can be configured to perform various computing operations, such as the various operations involved in machine learning in the field of artificial intelligence.

[0084] In different embodiments, the computing processing apparatus disclosed herein can be configured to perform user-specified operations. In exemplary applications, the computing processing apparatus can be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Thus, the operator code described above in conjunction with the accompanying drawings can be executed by the intelligent processor. Similarly, one or more computing devices included within the computing processing apparatus can be implemented as an artificial intelligence processor core or a portion of the hardware structure of an artificial intelligence processor core. When multiple computing devices are implemented as artificial intelligence processor cores or portions of the hardware structure of artificial intelligence processor cores, the computing processing apparatus disclosed herein can be considered to have a single-core structure or a homogeneous multi-core structure.

[0085] For example, the computing processing apparatus of this disclosure is as follows: Figure 8 As shown. According to the scheme of this disclosure, the computing processing device 800 can adopt a hierarchical design and can be implemented as a system-on-a-chip. Furthermore, it can include at least one cluster, and each cluster can include multiple processor cores. In other words, the computing processing device 800 is constructed in a hierarchical structure of system-on-a-chip, cluster, and processor cores.

[0086] From the perspective of system-on-a-chip hierarchy, such as Figure 8 As shown, the computing processing device 800 includes an external storage controller 81, a peripheral communication module 82, an on-chip interconnect module 83, a synchronization module 84, and multiple clusters 85. There can be multiple external storage controllers 81; two are shown exemplarily in the figure. These controllers respond to access requests from the processor core to access external storage devices, such as DRAM 408, to read data from or write data to external storage. The on-chip interconnect module 83 connects the external storage controller 81, the peripheral communication module 82, and the multiple clusters 85 to transmit data and control signals between the modules. The synchronization module 84 is a global barrier controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 85 are the computing cores of the multi-core processing device 800; four are shown exemplarily in the figure. With hardware development, the multi-core processing device 800 of this disclosure can also include 8, 16, 64, or even more clusters 85. The clusters 85 are used to efficiently execute deep learning algorithms.

[0087] From the perspective of cluster hierarchy, such as Figure 8As shown in the upper right corner, each cluster 85 includes a processing unit 802 and a memory core (MEM core) 804. The processing unit 802 performs various computational tasks. In some implementations, the processing unit may be a multi-core architecture, for example, including multiple processing cores (IPU cores) 811-1 to 811-n, to perform tasks such as large-scale vector computation. This disclosure does not limit the number of processing cores 811.

[0088] The internal architecture of the 811 core is as follows Figure 8 As shown below, each processing core 811 may have multiple computing modules 824-1 to 824-m for performing computing tasks, as well as a local storage module 823 required for performing computing tasks. It should be noted that the local storage module 823 may include various communication modules for exchanging data with external storage units. For example, the local storage module 823 may include a communication module 821 for communicating with the shared storage module 815 in the storage core 804. The communication module 821 may, for example, be a move direct memory access (MVDMA) module. The local storage module 823 may also include a communication module 822 for exchanging data with off-chip memory, such as DRAM 408. The communication module 822 may, for example, be an input / output direct memory access (IODMA) module. IODMA 822 controls memory access to NRAM / WRAM and DRAM 408 in local storage module 823; MVDMA 821 controls memory access to NRAM / WRAM and shared storage module 815 in local storage module 823.

[0089] continue Figure 8 In the upper right view, storage core 804 is mainly used for storage and communication, namely storing shared data or intermediate results between processing cores 811, and performing communication between cluster 85 and DRAM 408, communication between clusters 85, and communication between processing cores 811. In other embodiments, storage core 804 has scalar operation capabilities to perform scalar operations to implement computational tasks in data communication.

[0090] The storage core 804 includes a large shared memory module (SRAM) 815, a broadcast bus 814, a cluster direct memory access (CDMA) module 818, a global direct memory access (GDMA) module 816, and a communication-time computation module 817. The SRAM 815 acts as a high-performance data relay station. Data multiplexed between different processing cores 811 within the same cluster 85 can be relayed between processing cores 811 via the SRAM 815 instead of each processing core 811 obtaining data from the DRAM 408 individually. Therefore, the storage core 804 only needs to quickly distribute multiplexed data from the SRAM 815 to multiple processing cores 811, thereby improving inter-core communication efficiency and significantly reducing on-chip and off-chip I / O access.

[0091] Broadcast bus 814, CDMA 818, and GDMA 816 are used to perform communication between processing cores 811, communication between clusters 85, and data transfer between cluster 85 and DRAM 808, respectively. These will be explained below.

[0092] The broadcast bus 814 is used to complete high-speed communication between the processing cores 811 within the cluster 85. In this embodiment, the broadcast bus 814 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point (e.g., data transmission from one processing core to another) data transmission. Multicast is a communication method that transmits a copy of data from SRAM 815 to several specific processing cores 811. Broadcast is a communication method that transmits a copy of data from SRAM 815 to all processing cores 811, and is a special case of multicast.

[0093] GDMA 816 works in conjunction with external memory controller 81 to control memory access from SRAM 815 to DRAM 808 in cluster 85, or to read data from DRAM 808 into SRAM 815. As described above, communication between DRAM 808 and NRAM / WRAM in local memory module 823 can be achieved through two channels. The first channel is a direct connection between DRAM 808 and local memory module 823 via IODMA 822; the second channel involves first transmitting data between DRAM 808 and SRAM 815 via GDMA 816, and then transmitting data between SRAM 815 and local memory module 823 via MVDMA 821. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is significantly greater than that of the first channel. Therefore, communication between DRAM 808 and local memory module 823 may be more efficient via the second channel. Embodiments of this disclosure can select the data transmission channel based on the hardware conditions.

[0094] In some embodiments, storage core 804 can serve as a cache layer within cluster 85, significantly expanding communication bandwidth. Furthermore, storage core 804 can also facilitate communication with other clusters 85. For example, storage core 804 can implement communication functions such as broadcast, scatter, gather, reduce, and all-reduce between clusters 85. Broadcast refers to distributing the same data to all clusters; scatter refers to distributing different data to different clusters; gather refers to aggregating data from multiple clusters; reduce refers to processing data from multiple clusters according to a specified mapping function to obtain the final result and sending it to a specific cluster; and the difference between all-reduce and scatter is that the latter only sends the final result to one cluster, while all-reduce sends it to all clusters.

[0095] The communication-time computing module 817 can be used to complete computational tasks in communication processes, such as those involving the aforementioned protocols and full protocols, without relying on the processing unit 802, thereby improving communication efficiency and achieving the effect of "in-memory computing". Depending on different hardware implementations, the communication-time computing module 817 and the shared storage module 815 can be integrated in the same or different components. This disclosure embodiment is not limited in this respect, and as long as the functions implemented and the technical effects achieved are similar to those disclosed herein, they are all within the protection scope of this disclosure.

[0096] Further as Figure 8As shown, multiple processor cores form four processor clusters. Each processor cluster may include multiple processor cores, all of which can access and store data in the shared memory module SRAM815. The processor cores in each cluster can also access and store data in external DRAM memory located outside the processing device.

[0097] In exemplary operation, the computing processing device disclosed herein can interact with other processing devices through an interface device to jointly complete user-specified operations. Depending on the implementation, the other processing devices disclosed herein may include one or more types of processors such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and an artificial intelligence processor, both general-purpose and / or special-purpose processors. These processors may include, but are not limited to, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing processing device disclosed herein can be considered to have a single-core structure or a homogeneous multi-core structure. However, when the computing processing device and other processing devices are considered together, they can be considered to form a heterogeneous multi-core structure.

[0098] In one or more embodiments, the other processing device may serve as an interface between the computing processing device disclosed herein (which may be specifically embodied in artificial intelligence, such as neural network operations) and external data and control, performing basic controls including but not limited to data transfer, starting and / or stopping the computing device. In another embodiment, the other processing device may also cooperate with the computing processing device to jointly complete computational tasks.

[0099] In one or more embodiments, the interface device can be used to transfer data and control instructions between a computing processing device and other processing devices. For example, the computing processing device can obtain input data from other processing devices via the interface device and write it to on-chip storage (or memory) of the computing processing device. Further, the computing processing device can obtain control instructions from other processing devices via the interface device and write them to on-chip control cache of the computing processing device. Alternatively or optionally, the interface device can also read data from the storage device of the computing processing device and transmit it to other processing devices. In some scenarios, the interface device can also be implemented as an application programming interface between the computing processing device and other processing devices, including, for example, a driver interface, to transfer various instructions and programs to be executed by the computing processing device between them.

[0100] Additionally or optionally, the combined processing apparatus disclosed herein may further include a storage device. As shown in the figures, the storage device is connected to both the computing processing device and the other processing device. In one or more embodiments, the storage device may be used to store data from the computing processing device and / or the other processing device. For example, the data may be data that cannot be fully stored in the internal or on-chip storage of the computing processing device or other processing device.

[0101] In some embodiments, this disclosure also discloses a chip (e.g. Figure 12 The chip shown is 1202. In one implementation, this chip is a system-on-a-chip (SoC). This chip can be externally interfaced with devices such as... Figure 12 The external interface device 1206 shown is connected to other related components. These related components may be, for example, a camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. In some applications, the chip may integrate other processing units (e.g., video codecs) and / or interface modules (e.g., DRAM interfaces). In some embodiments, this disclosure also discloses a chip package structure that includes the aforementioned chip. In some embodiments, this disclosure also discloses a board that includes the aforementioned chip package structure. The following will be combined with... Figure 12 This board is described in detail.

[0102] Figure 12 This is a schematic diagram illustrating the structure of a board 1200 according to an embodiment of this disclosure, which may include the intelligent processor architecture described in conjunction with the accompanying drawings. Figure 12As shown, the board includes a storage device 1204 for storing data, which includes one or more storage cells 1210. This storage device can be connected and transmit data with the controller 1208 and the aforementioned chip 1202 via, for example, a bus. Furthermore, the board also includes an external interface device 1206, configured for data relay or switching between the chip (or a chip in a chip package) and an external device 1212 (e.g., a server or computer). For example, data to be processed can be transferred from the external device to the chip via the external interface device. Alternatively, the calculation results of the chip can be transmitted back to the external device via the external interface device. Depending on the application scenario, the external interface device can have different interface forms, such as a standard PCIe interface.

[0103] In one or more embodiments, the controller in the disclosed board can be configured to regulate the state of the chip. Therefore, in one application scenario, the controller may include a microcontroller (MCU) for regulating the operating state of the chip.

[0104] Based on the above combination Figure 11 and Figure 12 Based on the description, those skilled in the art will understand that this disclosure also discloses an electronic device or apparatus that may include one or more of the aforementioned boards, one or more of the aforementioned chips, and / or one or more of the aforementioned combined processing apparatus.

[0105] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0106] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0107] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document divides them based on logical functions, but in actual implementation, there may be other division methods. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0108] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0109] In some implementation scenarios, the integrated unit described above can be implemented as a software program module. If implemented as a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable storage device (CMSDD). Therefore, when the disclosed solution is embodied in a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, server, or network device) to execute some or all of the steps of the method described in the embodiments of this disclosure. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0110] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0111] The foregoing can be better understood in accordance with the following terms:

[0112] Clause A1, a processing apparatus comprising:

[0113] An arithmetic unit, configured to perform at least one arithmetic operation to obtain a result;

[0114] A first-type converter, configured to convert the data type of the operation result into a third data type;

[0115] Wherein, the data precision of the data type of the operation result is greater than the data precision of the third data type, and the third data type is suitable for storing and transporting the operation result.

[0116] Clause A2, the processing apparatus according to Clause A1, wherein the arithmetic unit comprises:

[0117] A first arithmetic unit, configured to perform a first type operation on a first data type, in order to obtain the result of the first type operation;

[0118] The second arithmetic unit, configured for:

[0119] Perform a second type operation on the result of the first type operation using a second data type, to obtain the result of the second type operation; and

[0120] The nonlinear layer operation of the neural network is performed on the result of the second type of operation to obtain the nonlinear layer operation result of the second type of operation.

[0121] The first type of converter is also configured to convert the result of the nonlinear layer operation into the result of the operation of the third data type.

[0122] Clause A3. The processing apparatus according to Clause A2, wherein the first data type has a low bit length data precision, the second data type has a high bit length data precision, and the third data type has a data precision less than the data precision of the first data type and / or the data precision of the second data type.

[0123] Clause A4. The processing apparatus according to Clause A3, wherein the first data type includes a half-precision floating-point data type, the second data type includes a single-precision floating-point data type, and the third data type includes a TF32 data type having a 10-bit mantissa and an 8-bit exponent.

[0124] Clause A5, pursuant to Clause A1, the first type converter is also configured for data type conversion between different arithmetic operations.

[0125] Clause A6. The processing apparatus according to Clause A1 further includes:

[0126] The second type of converter is configured to convert the result of an operation on a third data type into the first or second data type, so as to facilitate subsequent operations by the first or second arithmetic unit.

[0127] Clause A7. The processing apparatus according to Clause A6, wherein the first type converter and / or the second type converter are configured to truncate the operation results according to a nearest neighbor truncation method or a preset truncation method, so as to realize the conversion between data types.

[0128] Clause A8. The processing apparatus according to Clause A1 further includes:

[0129] At least one on-chip memory configured to store the results of operations on the third data type and to interact with data of the third data type in at least one off-chip memory.

[0130] Clause A9. The processing apparatus according to Clause A1 further includes:

[0131] A compressor configured to compress the results of operations on the third data type for storage and transport.

[0132] Clause A10, the processing apparatus according to any one of Clauses A6-9, wherein one or more of the first arithmetic unit, the second arithmetic unit, the first type converter, and the second type converter are configured to perform one or more of the following operations: operations on output neurons of a neural network inference process; operations on gradient propagation during neural network training; and operations on weight updates during neural network training.

[0133] Clause A11, the processing apparatus according to Clause A10, wherein in the neural network inference process and / or neural network training process, the first type of operation includes multiplication, the second type of operation includes addition, and the nonlinear layer operation includes activation.

[0134] Clause A12, an edge device for neural network operations, comprising a processing means according to any one of clauses A1-11, and configured to participate in the execution of training and / or inference operations of the neural network at the edge device.

[0135] Clause A13. A cloud device for neural network computation, comprising a processing means as described in any one of Clauses A1-11, and configured to participate in the execution of training and / or inference computations of the neural network at the cloud device.

[0136] Clause A14, a cloud-edge collaborative computing neural network system, comprising:

[0137] The cloud computing subsystem is configured to perform neural network-related operations in the cloud.

[0138] An edge computing subsystem, configured to perform neural network-related operations at the edge; and

[0139] The processing apparatus according to any one of clauses A1-11, wherein the processing apparatus is disposed at the cloud computing subsystem and / or the edge computing subsystem and is configured to participate in the execution of the training process of the neural network and / or the inference process based on the neural network.

[0140] Clause A15. A method for neural network computation, implemented by a processing device, and comprising: performing at least one computation operation to obtain a computation result; converting the data type of the computation result into a third data type; wherein the data precision of the data type of the computation result is greater than the data precision of the third data type, and the third data type is suitable for storing and transporting the computation result.

[0141] Clause A16. A computer program product comprising a computer program that, when executed by a processor, implements the method described in accordance with Clause A15.

[0142] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0143] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0144] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0145] While numerous embodiments of this disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and intent of this disclosure. It should be understood that various alternatives to the embodiments of this disclosure described herein may be employed in the practice of this disclosure. The appended claims are intended to define the scope of this disclosure and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A processing apparatus, comprising: An arithmetic unit configured to perform at least one arithmetic operation to obtain an arithmetic result, wherein the at least one arithmetic operation is used to perform an operation on data of a first data type and / or a second data type to obtain the arithmetic result; A first-type converter is configured to convert the data type of the operation result into a third data type. The first-type converter is configured to truncate the operation result according to the nearest neighbor principle or a preset truncation method to achieve the conversion between data types. Wherein, the data precision of the data type of the operation result is greater than the data precision of the third data type, the third data type is suitable for storing and transporting the operation result, and the data precision of the third data type is less than the data precision of the first data type and / or the data precision of the second data type; The first data type includes a half-precision floating-point data type, the second data type includes a single-precision floating-point data type, and the third data type includes a TF32 data type, wherein the TF32 data type has a 10-bit mantissa and an 8-bit exponent; wherein the arithmetic unit includes: A first arithmetic unit, configured to perform a first type operation on a first data type, in order to obtain the result of the first type operation; The second arithmetic unit, configured for: Perform a second type operation on the result of the first type operation using a second data type, to obtain the result of the second type operation; and The nonlinear layer operation of the neural network is performed on the result of the second type of operation to obtain the nonlinear layer operation result of the second type of data. The first type of converter is also configured to convert the result of the nonlinear layer operation into the result of the operation of the third data type.

2. The processing apparatus according to claim 1, wherein the first data type has low bit length data precision and the second data type has high bit length data precision.

3. The processing apparatus according to claim 1, wherein the first type converter is further configured for data type conversion between different arithmetic operations.

4. The processing apparatus according to claim 1, further comprising: The second type of converter is configured to convert the result of the operation of the third data type into the first data type or the second data type, so as to facilitate subsequent operations of the first arithmetic unit or the second arithmetic unit.

5. The processing apparatus according to claim 4, wherein the second type converter is configured to truncate the calculation result according to the nearest neighbor principle or a preset truncation method to achieve data type conversion.

6. The processing apparatus according to claim 1, further comprising: At least one on-chip memory configured to store the results of operations on the third data type and to interact with at least one off-chip memory using the third data type.

7. The processing apparatus according to claim 1, further comprising: A compressor configured to compress the results of operations on the third data type for storage and transport.

8. The processing apparatus according to any one of claims 4-5, wherein one or more of the first arithmetic unit, the second arithmetic unit, the first type converter, and the second type converter are configured to perform one or more of the following operations: Operations on the output neurons of a neural network during the inference process; This refers to the gradient propagation operation during neural network training; and This refers to the operation of updating weights during the training of a neural network.

9. The processing apparatus according to claim 8, wherein in the neural network inference process and / or neural network training process, the first type of operation includes multiplication, the second type of operation includes addition, and the nonlinear layer operation includes activation.

10. An edge device for neural network operations, comprising a processing means according to any one of claims 1-9, and configured to participate in the execution of training and / or inference operations of the neural network at the edge device.

11. A cloud device for neural network computation, comprising a processing means according to any one of claims 1-9, and configured to participate in the execution of training and / or inference computations of the neural network at the cloud device.

12. A cloud-edge collaborative computing neural network system, comprising: The cloud computing subsystem is configured to perform neural network-related operations in the cloud. An edge computing subsystem, configured to perform neural network-related operations at the edge; as well as The processing apparatus according to any one of claims 1-9, wherein the processing apparatus is disposed at the cloud computing subsystem and / or the edge computing subsystem, and is configured to participate in the execution of the training process of the neural network and / or the inference process based on the neural network.

13. A method for neural network computation, implemented by a processing device, and comprising: Perform at least one operation to obtain a result, wherein the at least one operation is used to perform an operation on data of a first data type and / or a second data type to obtain the result; The data type of the operation result is converted into a third data type; the operation result is truncated according to the nearest neighbor principle or a preset truncation method to achieve the conversion between data types; Wherein, the data precision of the data type of the operation result is greater than the data precision of the third data type, the third data type is suitable for storing and transporting the operation result, and the data precision of the third data type is less than the data precision of the first data type and / or the data precision of the second data type; The first data type includes a half-precision floating-point data type, the second data type includes a single-precision floating-point data type, and the third data type includes a TF32 data type, which has a 10-bit mantissa and an 8-bit exponent. Perform the first type operation on the first data type to obtain the result of the first type operation; Perform a second type operation on the result of the first type operation using a second data type, to obtain the result of the second type operation; and The nonlinear layer operation of the neural network is performed on the result of the second type of operation to obtain the nonlinear layer operation result of the second type of data. The result of the nonlinear layer operation is converted into the result of the operation of the third data type.

14. A computer program product comprising a computer program that, when executed by a processor, implements the method according to claim 13.

Images