Hardware implementation structure and method of neural network processor

Abstract

The invention relates to the technical field of computers, and particularly provides a hardware implementation structure and method of a neural network processor. The structure comprises: a prefetching module configured to obtain external data from an external memory based on data source information; the shared cache module is configured to be used for acquiring and caching upstream data from the upstream module; the format conversion module is configured to be used for receiving external data and upstream data, and performing format conversion on the external data and / or the upstream data based on the format conversion information to obtain format conversion data; the calculation module is configured to calculate the format conversion data based on the calculation information to obtain a calculation result; the multiplexing processing module is configured to process the calculation result based on the feature generation information to generate feature data; and if the current cycle processing frequency meets the target cycle processing frequency, outputting the feature data to a downstream module. According to the invention, the compatibility and flexibility of the neural network processor NPU can be improved, and the overhead problem of heterogeneous calculation is solved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and in particular, to a hardware implementation structure and method of a neural network processor. BACKGROUND

[0002] In the current chip ecosystem, the tight coupling of operators and hardware implementation structures is very common. In the process of artificial intelligence chip acceleration, most chips have done a lot of operator customization development work. There are great differences between different operators, and even the same kind of operator can be divided into multiple subcategories of operators.

[0003] In related technologies, the hardware of a neural network processor (NPU) is often implemented on an application-specific integrated circuit (ASIC) with pertinence, and it is difficult to simultaneously support multiple operators. Moreover, operator customization development not only greatly increases development costs, but also causes slow technical standardization process and compatibility problems. SUMMARY

[0004] The present disclosure is proposed in view of the above problems. The present disclosure provides a hardware implementation structure and method of a neural network processor.

[0005] According to one aspect of the present disclosure, a hardware implementation structure of a neural network processor is provided, configured to execute a target operator in a plurality of operators configured by a control module in a current loop processing number, the target configuration information of the target operator including data source information, format conversion information, calculation information and feature generation information; the hardware implementation structure comprises: a prefetch module configured to obtain external data of the current loop processing number from an external memory based on the data source information; a shared cache module configured to obtain and cache upstream data of the current loop processing number from an upstream module based on data source information; a format conversion module configured to receive the external data and the upstream data; perform format conversion on the external data and / or the upstream data based on the format conversion information to obtain format conversion data; a calculation module configured to perform calculation on the format conversion data based on the calculation information to obtain calculation results; a multiplexing processing module configured to process the calculation results based on the feature generation information to generate feature data; and if the current loop processing number meets a target loop processing number, output the feature data to a downstream module.

[0006] According to another aspect of the present disclosure, a hardware implementation method of a neural network processor is provided, applied to a hardware implementation structure of the neural network processor, the hardware implementation structure being configured to execute a target operator among a plurality of operators configured for a current loop processing number based on target configuration information of the target operator of the control module, the target configuration information including data source information, format conversion information, calculation information, and feature generation information; the hardware implementation method includes: a pre-fetch module configured to obtain external data for the current loop processing number from an external memory based on the data source information; a shared cache module configured to obtain and cache upstream data for the current loop processing number from an upstream module based on the data source information; a format conversion module configured to receive the external data and the upstream data, and perform format conversion on the external data and / or the upstream data based on the format conversion information to obtain format conversion data; a calculation module configured to perform calculation on the format conversion data based on the calculation information to obtain calculation results; a multiplexing processing module configured to perform processing on the calculation results based on the feature generation information to generate feature data, and output the feature data to a downstream module if the current loop processing number meets a target loop processing number.

[0007] As will be described in detail below, according to the hardware implementation structure of the neural network processor of the embodiments of the present disclosure, the pre-fetch module is configured to obtain external data for a current loop processing number from an external memory based on data source information; the shared cache module is configured to obtain and cache upstream data for the current loop processing number from an upstream module; the format conversion module is configured to receive the external data and the upstream data, and perform format conversion on the external data and / or the upstream data based on format conversion information to obtain format conversion data; the calculation module is configured to perform calculation on the format conversion data based on calculation information to obtain calculation results; the multiplexing processing module is configured to perform processing on the calculation results based on feature generation information to generate feature data; and if the current loop processing number meets a target loop processing number, the feature data is output to a downstream module, which can adapt to execute a plurality of operators with different calculation processes through different configuration information, adjust the structure internal path and data loop, improve the compatibility and flexibility of the neural network processor NPU, and at the same time, introduce the hardware structure in the NPU to solve the overhead problem of heterogeneous computing and release the load of the CPU.

[0008] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the subject technology. BRIEF DESCRIPTION OF DRAWINGS

[0009] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0010] Figure 1 The calculation range of various different types of operators in the related art is shown; Figure 2 A schematic diagram of a hardware implementation structure of a neural network processor provided by an exemplary embodiment of the present disclosure is shown; Figure 3 A structural schematic diagram of a format conversion module provided by an exemplary embodiment of the present disclosure is shown; Figure 4 A structural schematic diagram of a calculation module provided by an exemplary embodiment of the present disclosure is shown; Figure 5 A structural schematic diagram of a multiplexing processing module provided by an exemplary embodiment of the present disclosure is shown; Figure 6 A flowchart of a running mode of a hardware implementation structure of a neural network processor provided by an exemplary embodiment of the present disclosure is shown; Figure 7 A part of a flowchart of a hardware implementation structure running an RMS Norm operator provided by an exemplary embodiment of the present disclosure is shown; Figure 8 Another part of a flowchart of a hardware implementation structure running an RMS Norm operator provided by an exemplary embodiment of the present disclosure is shown; Figure 9 Still another part of a flowchart of a hardware implementation structure running an RMS Norm operator provided by an exemplary embodiment of the present disclosure is shown; Figure 10 Still another part of a flowchart of a hardware implementation structure running an RMS Norm operator provided by an exemplary embodiment of the present disclosure is shown; Figure 11 A flowchart of a hardware implementation method of a neural network processor provided by an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0011] In order to make the objectives, technical solutions and advantages of the present disclosure more apparent, the following will describe exemplary embodiments according to the present disclosure in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, and are not all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited by the exemplary embodiments described herein.

[0012] It should be understood that each of the steps recited in the method embodiments of the present disclosure can be performed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit performing the steps shown. The scope of the present disclosure is not limited in this regard.

[0013] The term "comprises" and variations thereof herein, do not necessarily mean "consist of. The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment." The term "another embodiment" means "at least one additional embodiment." The term "some embodiments" means "at least some embodiments." Related terms have analogous meanings. It is noted that the terms "first," "second," and the like, as used herein do not have any specific one-to-one correspondence with the terms "one," "two," etc., and are used to differentiate between different apparatuses, modules, or units, but do not otherwise limit the scope of the functions performed by these apparatuses, modules, or units.

[0014] It is noted that the terms "one," "multiple," and the like, as used herein, are illustrative and not limiting, and those skilled in the art will understand that "one" or "a" should be interpreted as "one or more" unless otherwise indicated in the context by clear language.

[0015] The names of the messages or information exchanged between the plurality of apparatuses in the embodiments of the present disclosure are used only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0016] With the continuous advancement of the artificial intelligence wave, various large models are emerging one after another, and various types of operators are the basis for the implementation of large models, and the number and types are also increasing. In the specific implementation of artificial intelligence operators, the current mainstream way is that hardware develops and optimizes different operators separately, and each operator corresponds to a hardware module / unit, which is selected for execution according to the actual running device at compile time or runtime, resulting in high operator development cost, high hardware migration cost, and other problems. With the iterative development of large models, artificial intelligence chips need to support as many operators as possible to support different large models, increasing the versatility of chip structures.

[0017] When a neural network processor runs different neural network models, the supported operators differ greatly, and the chip modules designed for them can only execute the corresponding operators. Moreover, the same operator also has great differences due to different processing methods, such as the Norm and Softmax operators involved in the normalization / activation operation in the Transformer model. According to the current common calculation method, the Norm operator can be divided into LayerNorm, Batch Norm, Instance Norm, Group Norm, and RMS Norm, etc. different types of operators according to the processing calculation dimension and range,Figure 1 The calculation ranges of various different types of operators in the related art are shown. Different types of operators have different requirements for the module structure, so the compatibility of the same structure for executing different neural network models is poor for the NPU.

[0018] For different Norm operators, the corresponding calculation formulas also differ. The implementation formulas of Layer Norm, Batch Norm, Instance Norm, and Group Norm are as follows: (1) Among them: denotes the normalization / activation result, denotes the input data, denotes the mean of the input data, denotes the variance of the input data, denotes the anti-0 factor, and denotes the vector of the input data in different dimensions.

[0019] The implementation formula of RMS Norm is as follows: (2) Among them, denotes the normalization / activation result, denotes the input data, denotes the total number of input data, denotes the rescale factor.

[0020] RMS Norm omits the calculation of the mean, so the formula is simplified. Batch Norm calculates the mean and variance for the entire dimension within a batch, so it can be obtained offline during inference, while the mean and variance of other Norms must be calculated online during inference. The data sources of the parameters in the inference settlement formula are shown in Table 1.

[0021] Table 1 Data sources of parameters in the inference settlement formula

[0022] Designing different hardware unit structures for different Norm operators would incur significant area costs. Furthermore, obtaining computational values ​​online often requires reading the data twice from Double Data Rate (DDR) synchronous dynamic random access memory: once specifically for calculating the mean and variance within the range, and the other time for calculating the actual normalized / activation values. This process incurs time costs associated with data reading and repeated processing, resulting in a performance penalty for the processor.

[0023] Therefore, in order to solve the above problems, this disclosure provides a hardware implementation structure that can support data calculation operations such as Softmax and Norm operators, such as normalization / activation calculation operations of Transformer models, thereby improving the hardware's compatibility with this type of operator in neural network models.

[0024] Figure 2 A schematic diagram of the hardware implementation structure of the neural network processor provided in an exemplary embodiment of this disclosure is shown. Figure 2 As shown in the embodiments of this disclosure, a hardware implementation structure 200 for a neural network processor is provided, which is used to execute a target operator based on the target configuration information corresponding to the target operator among a variety of operators configured by the control module in the current loop processing number. The target configuration information includes data source information, format conversion information, calculation information and feature generation information.

[0025] Here, the control module can be an external control module (generally called a CMD module). The operation mode of the entire hardware implementation structure 200 can be adjusted according to the configuration information controlled by the external control module and the control information configured by the command module.

[0026] The aforementioned operators may include the Norm and Softmax operators involved in the normalization / activation operations used in the Transformer model. The Norm operator may include Layer Norm, Batch Norm, Instance Norm, Group Norm, and RMS Norm, but is not limited to these. The operator may be selected according to actual needs, and this disclosure does not impose specific limitations on this.

[0027] like Figure 2 As shown, the hardware implementation structure 200 includes: The prefetch module 201 is configured to retrieve external data for the current loop processing number from external memory based on data source information; The shared cache module 205 is configured to obtain and cache upstream data for the current loop processing count from the upstream module; The format conversion module 202 is configured to receive external data and upstream data; and to convert the format of the external data and / or upstream data based on the format conversion information to obtain format-converted data. Calculation module 203 is configured to perform calculations on the format conversion data based on the calculation information to obtain the calculation results; The reuse processing module 204 is configured to process the calculation results based on feature generation information to generate feature data; if the current number of loop processing times meets the target number of loop processing times, the feature data is output to the downstream module.

[0028] Specifically, such as Figure 2 As shown, the hardware implementation structure 200 includes a prefetch module 201 (called the Pre_fetch module), a format conversion module 202 (called the Convert module), a calculation module 203 (called the Count module), and a multiplexing module 204 (called the Reduce module) connected in sequence, as well as a shared cache module 205 (called the Buffer module) connecting the format conversion module 202 and the calculation module 203.

[0029] like Figure 2 As shown, the hardware implementation structure 200 includes two input ports, which can receive input data from external memory DDR and upstream module; and two output ports, which can output data from multiplexing processing module 204 or shared buffer module 205. Here, the input data may include external data obtained by prefetch module 201 from external memory and upstream data received by shared buffer module 205 from upstream module.

[0030] The external data stream has four ports: upstream and downstream. Upstream, there are two input ports: one connects to the DDR and the internal Pre_fetch module for direct data reading from the DDR, and the other connects to the Buffer module for receiving data from the upstream module. Downstream, there are two output ports: one connects to the Reduce module for direct output of the final calculation results (i.e., feature data), and the other connects to the Buffer module for sending intermediate results or data to be stored to the downstream module.

[0031] The Pre_fetch module is responsible for interacting with external storage, primarily by interacting with DDR based on data source information. Since reading data from DDR takes a long time, to improve efficiency, the Pre_fetch module prefetches data for the next computation cycle within the current computation cycle, thereby increasing data throughput and operational efficiency. Here, the data source information can be used to dynamically determine the source of the external data stream to execute different operators, and these operators have different data flow graphs.

[0032] The Buffer module primarily serves as a shared storage function. It can directly receive upstream data from upstream modules, and can also interact with the Convert and Count modules. It can function as an input / output data port, store intermediate calculation results and share them with other modules, and send the final calculation results to downstream modules.

[0033] The Convert module can receive external data from external storage processed by the Pre_fetch module, and also receive upstream data from the Buffer module. The Convert module primarily performs data format conversion, transforming the received external and / or upstream data based on format conversion information to obtain format-converted data. Here, the aforementioned format conversion information can be used to determine the target format so that the format-converted data meets the target format. The Convert module can also directly send the format-converted data to the downstream Count module, or temporarily store the format-converted data in the Buffer module.

[0034] The Count module can receive format-converted data from the Convert module, as well as format-converted data from the Buffer module. It can also simultaneously receive feature data returned from the Reduce module for computation. The Count module primarily handles internal data computation, performing calculations on the format-converted data based on computational information to obtain the results. Here, the computational information can be used to determine the target computation method, ensuring that the resulting computation method satisfies the target computation method. The Count module can also directly send the computation results to the downstream Reduce module, or temporarily store the results in the Buffer module.

[0035] The Reduce module primarily handles the generation of feature data within the structure. It processes the calculation results based on feature generation information to generate feature data. Here, the feature generation information can be used to determine the target feature generation method, ensuring that the resulting feature data's feature generation method satisfies the target feature generation method. The Reduce module can also output feature data to downstream modules if the current loop processing count meets the target loop processing count, or it can return the feature data to the Count module to continue participating in the calculation. Here, the target loop processing count can be determined according to actual needs, and this embodiment does not impose a specific limitation on it.

[0036] Based on this, the hardware implementation structure of the neural network processor provided in this disclosure can execute target operators among multiple operators based on the target configuration information configured by the control module. This allows the same hardware implementation structure to adapt to various operators with different computational processes by adjusting the internal pathways and data loops through different configuration information, thereby improving the compatibility and flexibility of the neural network processor (NPU). Simultaneously, by utilizing a shared buffer module as an internal data transfer station, a multi-data-path data processing hardware structure capable of multiple internal loop calculations is implemented, and this structure supports online acquisition of feature data within data batches. Furthermore, introducing this hardware structure within the NPU can solve the overhead problem of heterogeneous computing and reduce the CPU load.

[0037] In some embodiments, format conversion information is used to indicate the target formatting path corresponding to the target format. Figure 3 A schematic diagram of the structure of the format conversion module provided in an exemplary embodiment of this disclosure is shown, such as... Figure 3 As shown, the format conversion module includes a parallel formatting bypass path and multiple formatting paths, including a quantization path, an inverse quantization path, an exponentialization path, and a bias path. The format conversion module is also configured to, if the external data and / or upstream data do not meet the target format, determine the formatting path corresponding to the target format among multiple formatting paths as the target formatting path; and use the target formatting path to convert the format of the external data and / or upstream data to obtain format-converted data. If the external data and / or upstream data meet the target format, the formatting bypass path is determined as the target formatting path, and the external data and / or upstream data are determined as format conversion data; the format conversion data is sent to the computing module through the target formatting path.

[0038] Specifically, such as Figure 3 As shown, the Convert module includes parallel quantization paths (Quant), dequantization paths (Dequant), indexation paths (Indexation), and bias paths (Bias). These paths can perform data conversions between different formats of received external and / or upstream data, including unsigned numbers, fixed-point and floating-point numbers, and fixed-point numbers of different ranges, facilitating subsequent computational operations.

[0039] According to the embodiments of this disclosure, a formatting path can be selected to perform format conversion on external data and / or upstream data based on the target formatting path corresponding to the target format indicated by the format conversion information, so that the obtained format conversion data meets the target format indicated by the format conversion information.

[0040] The Convert module also includes a formatting bypass path, Bypass1, through which external data and / or upstream data that do not require data format changes can be directly processed.

[0041] The Convert module has parallel pathways that can be expanded for functionality. It can receive data from external storage processed by the Pre_fetch module, as well as data from the Buffer module. Processed data can be directly sent to the downstream Count module or temporarily stored in the Buffer module. Data gating is uniformly controlled by CMD based on configuration information. Convert can temporarily store format-converted data in the Buffer module and simultaneously send the converted data to the downstream calculation module. The Buffer module can send upstream received data or partially converted data back to the Convert module for multiple data format processing steps.

[0042] In some embodiments, computational information is used to indicate a target computational path corresponding to a target computational method. Figure 4 A schematic diagram of the structure of the computing module provided in an exemplary embodiment of this disclosure is shown, such as... Figure 4 As shown, the calculation module includes parallel calculation bypass paths and multiple calculation paths, including multiplication path, reciprocal calculation path, addition path and reciprocal square root path; The calculation module is also configured to, if there is a calculation requirement for the format conversion data, identify the calculation path corresponding to the target calculation method among multiple calculation paths as the target calculation path; and use the target calculation path to perform calculations on the format conversion data to obtain the calculation results. If the format conversion data does not require computation, the computation bypass path is determined as the target computation path, and the format conversion data is determined as the computation result; the computation result is sent to the multiplexing processing module through the target computation path.

[0043] Specifically, such as Figure 4 As shown, the Count module includes parallel multiplication paths Mul / Fp_mul, reciprocal calculation path Rcp, addition path Add / FP_add, and reciprocal square root path Rsq. These paths can provide calculation operations for data in different formats. In this embodiment, the calculation information can be used to indicate the target calculation path corresponding to the target calculation method, and a calculation path can be selected to calculate the format-converted data, so that the calculation method of the obtained calculation result satisfies the target calculation method indicated by the calculation information.

[0044] The Count module also includes a bypass path, Bypass2, which allows data without computational requirements to be processed directly through this path.

[0045] The Count module has parallel processing paths, enabling multi-path parallel computation. Its functionality is expandable; it can receive data processed by the Convert module, data from the Buffer module, and feature data returned from the Reduce module for computation. It can directly send processed data or upstream data to the downstream Reduce module, or temporarily store processed data in the Buffer module. The Buffer module can send upstream data or partially processed data back to the Count module for further computation. The Buffer module also receives some directly compulsible results from the Count module and sends them downstream.

[0046] In some embodiments, feature generation information is used to indicate the target feature generation path corresponding to the target feature generation method. Figure 5 A schematic diagram of the structure of the multiplexing processing module provided in an exemplary embodiment of this disclosure is shown, such as... Figure 5 As shown, the multiplexing processing module includes parallel feature generation bypass paths and multiple feature generation paths, including a maximum value calculation path and a summation or mean calculation path. The reuse processing module is also configured to, if the calculation results require feature generation, identify the feature generation path that corresponds to the target feature generation method among multiple feature generation paths as the target feature generation path; and use the target feature generation path to process the calculation results to obtain feature data. If the calculation result does not require feature generation, the feature generation bypass path is determined as the target feature generation path, and the calculation result is determined as feature data; if the current loop processing number meets the target loop processing number, the feature data is output to the downstream module through the target feature generation path.

[0047] Specifically, such as Figure 5 As shown, the multiplexing processing module includes a parallel maximum value calculation path (MAX) and a summation or mean value calculation path (Mean / Sum). These paths can provide different feature data generation operations. In this embodiment, feature generation information can be used to indicate the target feature generation path corresponding to the target feature generation method. A feature generation path can then be selected to generate features from the calculation results, ensuring that the feature generation method of the obtained feature data satisfies the target feature generation method indicated by the feature generation information.

[0048] The Reduce module also includes a feature generation bypass path, Bypass3, which allows data without feature generation requirements to be processed directly through this path.

[0049] The Reduce module has parallel pathways and can be expanded to include features. It can receive data from the Count module and return feature data to the Count module, or it can act as an output module to send data from the Count module to downstream modules without any processing.

[0050] In some embodiments, Figure 6 A flowchart illustrating the operation of the hardware implementation structure of the neural network processor provided in the exemplary embodiments of this disclosure is shown, such as... Figure 6 As shown, the hardware implementation structure is also used to update the target configuration information to the configuration information of the target operator reconfigured by the control module if the current number of loop processing does not meet the target number of loop processing.

[0051] Specifically, the first step is to determine the operator functions that the hardware implementation of the neural network processor will carry in the current loop processing iteration, i.e., to determine the target operator. The CMD (Controlled Decision Machine) sends the target control information corresponding to the current operator configuration to the hardware implementation. This target control information includes configuration information for path selection and data format conversion (i.e., the data source information, format conversion information, computation information, and feature generation information mentioned earlier). The target configuration information is cached and stored in the internal control information register of the hardware implementation. Based on the target configuration information, the hardware implementation determines the upstream data source and the internal pathways and bypass paths, performs the first round of data processing, and then determines whether loop path processing is needed based on the operator processing stage. The CMD subsequently modifies the configuration information, updates the configuration stored within the structure, re-determines the data processing path within the structure, and performs another judgment until the data processing is complete and outputs to the downstream module.

[0052] Taking the processing of the RMS Norm operator as an example, the calculation formula can be changed as follows: (3) (4) Among them, input data Provided by the upstream module, Read directly from DDR.

[0053] Figure 7 This illustration shows a portion of the process flow of the hardware implementation structure providing an exemplary embodiment of the present disclosure running the RMS Norm operator, as shown below. Figure 7 As shown, received from the upstream module Data is read into the Convert module through the Buffer path for quantization format conversion. After format conversion, One copy is temporarily stored in the Buffer module for later processing, and another copy is sent to the Count module. The copy sent to the Count module... Perform a multiplication operation on the data to obtain The data is sent to the Reduce module, where a summation operation is performed to obtain... data.

[0054] Figure 8 This illustration shows another part of the flowchart of the hardware implementation structure providing an exemplary embodiment of the present disclosure running the RMS Norm operator, as shown below. Figure 8 As shown, reading from DDR Data is transmitted from the Pre_fetch module to the Convert module, then through the Bypass path within the Convert module, and finally to the Count module. Simultaneously, converted data is read from the Buffer module. xi In the Count module, perform a multiplication operation to obtain... The data is temporarily stored in the Buffer module.

[0055] Figure 9 This illustration shows another part of the flowchart of the hardware implementation structure provided in the exemplary embodiments of this disclosure running the RMS Norm operator, as shown below. Figure 9 As shown, the Reduce module will obtain... The data is returned to the Count module, which performs the reciprocal square root operation to obtain the result. The data is then sent to the Buffer module for temporary storage.

[0056] Figure 10 This illustration shows another part of the flowchart of the hardware implementation structure provided in the exemplary embodiments of this disclosure running the RMS Norm operator, as shown below. Figure 10 As shown, read the temporarily stored data from the Buffer module. Data and The data is fed into the Count module, where a multiplication operation is performed to obtain the result. Data, i.e., the final result It is then sent to the Reduce module, and then to the downstream module via the Bypass path. This completes the execution of the RMS Norm operator.

[0057] For the Softmax operator, which also requires obtaining the MAX feature value for calculation, the processing and execution method is similar to that of the Norm operator. Moreover, it only requires a single input port, and this hardware implementation structure can also be well compatible. This hardware implementation structure has good adaptability to various operators with similar processing methods that require online acquisition of data feature values.

[0058] Based on this, the hardware implementation structure of the neural network processor provided in this disclosure can support normalization / activation calculation operations such as Softmax and Norm within the Transformer model. This hardware implementation structure can be deployed within the NPU. When executing a large Transformer neural network model, it adapts to the execution of multiple normalization / activation operators by adjusting the internal pathways and data loops, improving the adaptability and compatibility of the NPU. This solves the problem in related technologies where customized artificial intelligence chips are difficult to simultaneously support multiple normalization / activation operators within the Transformer model. The multi-port, parallel arrangement of data pathways enhances the scalability of the structure. For calculations requiring more complex steps or procedures, expansion can be achieved simply by adding parallel processing pathways within the corresponding sub-modules.

[0059] This disclosure also provides a hardware implementation method for a neural network processor, applied to the hardware implementation structure of the aforementioned neural network processor. The hardware implementation structure is used to execute the target operator based on target configuration information corresponding to the target operator among a variety of operators configured by the control module in the current loop processing number. The target configuration information includes data source information, format conversion information, calculation information, and feature generation information.

[0060] Figure 11 A flowchart illustrating a hardware implementation method for a neural network processor provided in an exemplary embodiment of this disclosure is shown. Figure 11 As shown, the hardware implementation method of this neural network processor includes: S1101, the prefetch module retrieves external data for the current loop processing number from the external memory based on the data source information; S1102, the shared cache module obtains and caches upstream data for the current loop processing number from the upstream module based on the data source information; S1103, The format conversion module receives external data and upstream data; based on the format conversion information, it performs format conversion on the external data and / or upstream data to obtain format-converted data; S1104, The calculation module performs calculations on the format conversion data based on the calculation information to obtain the calculation results; S1105, the reuse processing module processes the calculation results based on the feature generation information to generate feature data; if the current number of loop processing times meets the target number of loop processing times, the feature data is output to the downstream module.

[0061] For details, please refer to the previous section on the hardware implementation structure of neural network processors; it will not be repeated here.

[0062] In some embodiments, format conversion information is used to indicate a target formatting path corresponding to a target format; the format conversion module includes a formatting bypass path and multiple formatting paths configured in parallel, the multiple formatting paths including a quantization path, an inverse quantization path, an exponentialization path, and a bias path; the method may further include: If the external data and / or upstream data do not meet the target format, the formatting path that corresponds to the target format among multiple formatting paths is determined as the target formatting path; the external data and / or upstream data are converted using the target formatting path to obtain format-converted data; If the external data and / or upstream data meet the target format, the formatting bypass path is determined as the target formatting path, and the external data and / or upstream data are determined as format conversion data; the format conversion data is sent to the computing module through the target formatting path.

[0063] For details, please refer to the previous section on the hardware implementation structure of neural network processors; it will not be repeated here.

[0064] In some embodiments, the calculation information is used to indicate a target calculation path corresponding to the target calculation method; the calculation module includes parallelly configured calculation bypass paths and multiple calculation paths, including multiplication paths, reciprocal calculation paths, addition paths, and reciprocal square root paths; the method may further include: If the format conversion data requires computation, the computation path that corresponds to the target computation method among multiple computation paths is identified as the target computation path; the target computation path is then used to perform computation on the format conversion data to obtain the computation result. If the format conversion data does not require computation, the computation bypass path is determined as the target computation path, and the format conversion data is determined as the computation result; the computation result is sent to the multiplexing processing module through the target computation path.

[0065] For details, please refer to the previous section on the hardware implementation structure of neural network processors; it will not be repeated here.

[0066] In some embodiments, feature generation information is used to indicate a target feature generation path corresponding to a target feature generation method; the multiplexing processing module includes parallel feature generation bypass paths and multiple feature generation paths, the multiple feature generation paths including a maximum value calculation path and a summation or mean calculation path; the method further includes: If the calculation results require feature generation, then the feature generation path that corresponds to the target feature generation method among multiple feature generation paths is determined as the target feature generation path; the calculation results are processed using the target feature generation path to obtain feature data; If the calculation result does not require feature generation, the feature generation bypass path is determined as the target feature generation path, and the calculation result is determined as feature data; if the current loop processing number meets the target loop processing number, the feature data is output to the downstream module through the target feature generation path.

[0067] For details, please refer to the previous section on the hardware implementation structure of neural network processors; it will not be repeated here.

[0068] In some embodiments, the method may further include: If the current number of iterations does not meet the target number of iterations, the target configuration information will be updated to the configuration information of the target operator reconfigured by the control module.

[0069] For details, please refer to the previous section on the hardware implementation structure of neural network processors; it will not be repeated here.

[0070] The above description is merely an illustration of some embodiments of this disclosure and the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0071] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A hardware implementation structure for a neural network processor, characterized in that, This is used to execute the target operator based on the target configuration information corresponding to the target operator among various operators configured by the control module in the current loop processing number. The target configuration information includes data source information, format conversion information, calculation information, and feature generation information. The hardware implementation structure includes: The prefetch module is configured to retrieve external data for the current loop processing number from an external memory based on the data source information; A shared cache module is configured to obtain and cache upstream data from the upstream module based on data source information for the current loop processing number; A format conversion module is configured to receive the external data and the upstream data; and to convert the format of the external data and / or the upstream data based on the format conversion information to obtain format-converted data. The calculation module is configured to perform calculations on the format conversion data based on the calculation information to obtain the calculation result; The reuse processing module is configured to process the calculation results based on the feature generation information to generate feature data; if the current number of loop processing times meets the target number of loop processing times, the feature data is output to the downstream module.

2. The hardware implementation structure of the neural network processor as described in claim 1, characterized in that, The format conversion information is used to indicate the target formatting path corresponding to the target format; the format conversion module includes a parallel formatting bypass path and multiple formatting paths, the multiple formatting paths including a quantization path, an inverse quantization path, an exponentialization path, and a bias path; The format conversion module is further configured to determine the formatting path corresponding to the target format among the multiple formatting paths as the target formatting path if the external data and / or the upstream data do not meet the target format. Using the target formatting path, the external data and / or the upstream data are format-converted to obtain the format-converted data; If the external data and / or the upstream data meet the target format, then the formatting bypass path is determined as the target formatting path, and the external data and / or the upstream data is determined as the format conversion data; the format conversion data is sent to the computing module through the target formatting path.

3. The hardware implementation structure of the neural network processor as described in claim 1, characterized in that, The calculation information is used to indicate the target calculation path corresponding to the target calculation method; the calculation module includes parallel calculation bypass paths and multiple calculation paths, the multiple calculation paths including multiplication path, reciprocal calculation path, addition path and reciprocal square root path; The calculation module is also configured to determine the calculation path corresponding to the target calculation method among the multiple calculation paths as the target calculation path if the format conversion data has a calculation requirement. The target computing path is used to perform calculations on the format-converted data to obtain the calculation results; If the format conversion data does not meet the computational requirement, then the computational bypass path is determined as the target computational path, and the format conversion data is determined as the computational result; The calculation results are sent to the multiplexing processing module through the target calculation path.

4. The hardware implementation structure of the neural network processor as described in claim 1, characterized in that, The feature generation information is used to indicate the target feature generation path corresponding to the target feature generation method; the multiplexing processing module includes a parallel feature generation bypass path and multiple feature generation paths, the multiple feature generation paths including a maximum value calculation path and a summation or average value calculation path; The reuse processing module is further configured to, if the calculation result has a feature generation requirement, determine the feature generation path that corresponds to the target feature generation method among the multiple feature generation paths as the target feature generation path; The target feature generation path is used to process the calculation results to obtain the feature data; If the calculation result does not meet the feature generation requirement, then the feature generation bypass path is determined as the target feature generation path, and the calculation result is determined as the feature data; if the current loop processing number meets the target loop processing number, the feature data is output to the downstream module through the target feature generation path.

5. The hardware implementation structure of the neural network processor as described in any one of claims 1 to 4, characterized in that, The hardware implementation structure is further configured to update the target configuration information to the configuration information of the target operator reconfigured by the control module if the current number of loop processing cycles does not meet the target number of loop processing cycles.

6. A hardware implementation method for a neural network processor, applied to the hardware implementation structure of the neural network processor as described in any one of claims 1 to 5, characterized in that, The hardware implementation structure is used to execute the target operator based on the target configuration information corresponding to the target operator among the various operators configured by the control module in the current loop processing number. The target configuration information includes data source information, format conversion information, calculation information and feature generation information. The hardware implementation method includes: The prefetch module retrieves the external data for the current loop processing count from the external memory based on the data source information. The shared cache module obtains and caches the upstream data for the current loop processing count from the upstream module based on the data source information; The format conversion module receives the external data and the upstream data; it performs format conversion on the external data and / or the upstream data based on the format conversion information to obtain format-converted data. The calculation module performs calculations on the format conversion data based on the calculation information to obtain the calculation results; The reuse processing module processes the calculation results based on the feature generation information to generate feature data; if the current number of loop processing times meets the target number of loop processing times, the feature data is output to the downstream module.

7. The hardware implementation method of the neural network processor as described in claim 6, characterized in that, The format conversion information is used to indicate the target formatting path corresponding to the target format; the format conversion module includes a parallel formatting bypass path and multiple formatting paths, the multiple formatting paths including a quantization path, an inverse quantization path, an exponentialization path, and a bias path; the method further includes: If the external data and / or the upstream data do not meet the target format, then the formatting path corresponding to the target format among the multiple formatting paths is determined as the target formatting path; the external data and / or the upstream data are format-converted using the target formatting path to obtain the format-converted data; If the external data and / or the upstream data meet the target format, then the formatting bypass path is determined as the target formatting path, and the external data and / or the upstream data is determined as the format conversion data; the format conversion data is sent to the computing module through the target formatting path.

8. The hardware implementation method of the neural network processor as described in claim 6, characterized in that, The calculation information is used to indicate the target calculation path corresponding to the target calculation method; the calculation module includes parallel calculation bypass paths and multiple calculation paths, including multiplication paths, reciprocal calculation paths, addition paths, and reciprocal square root paths; the method further includes: If the format conversion data requires computation, then the computation path corresponding to the target computation method among the multiple computation paths is determined as the target computation path; the format conversion data is calculated using the target computation path to obtain the computation result; If the format conversion data does not meet the computational requirement, then the computational bypass path is determined as the target computational path, and the format conversion data is determined as the computational result; the computational result is sent to the multiplexing processing module through the target computational path.

9. The hardware implementation method of the neural network processor as described in claim 6, characterized in that, The feature generation information is used to indicate the target feature generation path corresponding to the target feature generation method; the multiplexing processing module includes parallel feature generation bypass paths and multiple feature generation paths, the multiple feature generation paths including a maximum value calculation path and a summation or mean calculation path; the method further includes: If the calculation result requires feature generation, then the feature generation path that corresponds to the target feature generation method among the multiple feature generation paths is determined as the target feature generation path; the calculation result is processed using the target feature generation path to obtain the feature data; If the calculation result does not meet the feature generation requirement, then the feature generation bypass path is determined as the target feature generation path, and the calculation result is determined as the feature data; if the current loop processing number meets the target loop processing number, the feature data is output to the downstream module through the target feature generation path.

10. The hardware implementation method of the neural network processor as described in any one of claims 6 to 9, characterized in that, The method further includes: If the current number of iterations does not meet the target number of iterations, the target configuration information is updated to the configuration information of the target operator reconfigured by the control module.

Images