A large language model modular reasoning computing system, method, device and medium

Abstract

The present invention discloses a large language model modular reasoning computing system, method, device and medium. By configuring different units of the text embedding module, multi-cluster attention module, multi-layer perceptron module and linear mapping module in the large language model modular reasoning computing system, the calculation of the generated word token of the input text token sequence is realized. In addition, the attention mechanism calculation data flow method and the linear layer calculation data flow method are improved. The problem that the computing efficiency of the chip is difficult to give full play to, the performance is poor and the requirements for professionals are high in the prior art due to the limitation of external storage bandwidth is solved, and a more complete large language model is built, and a high-throughput computing data flow is designed to give full play to the computing power of the chip.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a large language model modular reasoning computing system, method, device and medium. Background Art

[0002] A large amount of intermediate data is generated during the inference calculation process of a large language model. This data is frequently transmitted between the chip and external storage, which significantly reduces the data throughput of the entire inference calculation process and increases system latency. Given that large language models contain complex computational logic, such as matrix calculations, activation functions, attention mechanisms, normalization, and residual calculations, the current known technology for large language model inference is to explicitly call these single-function operators to splice the computational logic of the entire network.

[0003] In the process of realizing the present invention, the inventors found that the prior art has the following defects: At present, generally multiple independent and single-function general operators are combined to perform reasoning calculations on large language models, and each operator needs to interact with external storage for data at the beginning and end of the calculation. The storage architecture of the chip generally adopts an internal multi-level cache method, and the bandwidth for data exchange with external storage is the lowest, and the higher the internal cache level, the greater the bandwidth. Therefore, the technical solution of a single operator combination is limited by the external storage bandwidth, and it will be difficult to give full play to the computing efficiency of the chip.

[0004] At the same time, there is room for direct data interaction or fusion of computing functions between some continuous operators in the internal cache of the chip, which can further improve computing efficiency. For the fusion between operators, there are solutions for automatic fusion through graph optimization and other technologies, but the main goal of this technology is to optimize general-type networks. Generally, fine-grained fusion strategies are used, which rely on fusion algorithms and require operator developers to develop corresponding adaptation functions. In the fusion of large language model networks, it is difficult to fully utilize their unique computing modes and the characteristics of the data themselves, and there is still a lot of room for performance improvement. On the other hand, the calculation of large language models is relatively complex, and developers need to have a deep understanding of each calculation and correctly call each single operator to build the network. This places relatively high demands on the professional quality of personnel and is not conducive to the rapid deployment of large language models. Summary of the invention

[0005] The present invention provides a large language model modular reasoning computing system, method, device and medium to achieve the construction of a more complete large language model, and designs a high-throughput computing data flow to fully utilize the computing power of the chip.

[0006] According to one aspect of the present invention, a large language model modular reasoning computing system is provided, wherein the large language model modular reasoning computing system includes a text embedding module, a multi-cluster attention module, a multi-layer perceptron module and a linear mapping module;

[0007] Wherein, the text embedding module includes a first configurable normalization calculation unit; the first configurable normalization calculation unit includes: a first non-normalization subunit, a first LayerNorm normalization calculation subunit and a first RMSNorm normalization calculation subunit;

[0008] The multi-cluster attention module includes a second configurable normalization calculation unit, a feature extraction calculation unit, a multi-cluster attention mechanism calculation unit, an output linear mapping calculation unit and a first configurable residual link calculation unit; wherein the second configurable normalization calculation unit includes: a second non-normalization subunit, a second LayerNorm normalization calculation subunit and a second RMSNorm normalization calculation subunit; the first configurable residual link calculation unit includes a first non-residual link calculation subunit and a first residual link calculation subunit;

[0009] The multi-layer perceptron module includes a third configurable normalization calculation unit, a linear dimension increase calculation unit, a configurable activation function calculation unit, a linear dimension reduction calculation unit and a second configurable residual link calculation unit; wherein the third configurable normalization calculation unit includes: a third non-normalization subunit, a third LayerNorm normalization calculation subunit and a third RMSNorm normalization calculation subunit; the configurable activation function calculation unit includes a calculation unit of at least one activation function type; the second configurable residual link calculation unit includes a second non-residual link calculation subunit and a second residual link calculation subunit;

[0010] The linear mapping module includes a fourth configurable normalization calculation unit and a linear probability mapping calculation unit; wherein the fourth configurable normalization calculation unit includes: a fourth non-normalization subunit, a fourth LayerNorm normalization calculation subunit and a fourth RMSNorm normalization calculation subunit;

[0011] Wherein, the multi-cluster attention mechanism calculation unit is calculated based on an improved attention mechanism calculation data flow method;

[0012] The feature extraction calculation unit, the output linear mapping calculation unit, the feature extraction calculation unit, the output linear mapping calculation unit and the linear probability mapping calculation unit are all calculated based on an improved linear layer calculation data flow method.

[0013] According to another aspect of the present invention, a large language model modular reasoning calculation method is provided, wherein the method is performed by a large language model modular reasoning calculation system, and includes:

[0014] Acquire an input text token sequence to be calculated in real time, perform feature vector search processing on the input text token sequence to be calculated through a text embedding module, and perform configurable normalization processing through a first configurable normalization calculation unit to obtain a feature vector of the input text token sequence;

[0015] Input the input text token sequence feature vector into the multi-layer loop corresponding to the large language model, and loop through the multi-cluster attention module and the multi-layer perceptron module layer by layer until all layers of loops are traversed to obtain the linear calculation result of the input text token sequence;

[0016] Inputting the linear calculation result of the input text token sequence into the linear mapping module for token probability mapping processing to obtain the generated word tokens of each input text;

[0017] The word tokens generated from each input text are fed back to the user.

[0018] According to another aspect of the present invention, a large language model modular reasoning computing device is provided, wherein the large language model modular reasoning computing system is executed, and includes:

[0019] An input text token sequence feature vector determination module is used to obtain the input text token sequence to be calculated in real time, perform feature vector search processing on the input text token sequence to be calculated through the text embedding module, and perform configurable normalization processing through the first configurable normalization calculation unit to obtain the input text token sequence feature vector;

[0020] The linear calculation result determination module of the input text token sequence is used to input the feature vector of the input text token sequence into the multi-layer loop corresponding to the large language model, and loop between the multi-cluster attention module and the multi-layer perceptron module layer by layer until each layer of loop is traversed to obtain the linear calculation result of the input text token sequence;

[0021] Each input text generated word token determination module is used to input the linear calculation result of the input text token sequence into the linear mapping module for token probability mapping processing to obtain each input text generated word token;

[0022] Each input text generates a word token feedback module, which is used to feed back the word token generated by each input text to the user.

[0023] According to another aspect of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the large language model modular reasoning calculation method described in any embodiment of the present invention is implemented.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the large language model modular reasoning calculation method described in any embodiment of the present invention when executed.

[0025] The technical solution of the embodiment of the present invention, the large language model modular reasoning computing system includes a text embedding module, a multi-cluster attention module, a multi-layer perceptron module and a linear mapping module; wherein the text embedding module includes a first configurable normalization computing unit; the first configurable normalization computing unit includes: a first non-normalization subunit, a first LayerNorm normalization computing subunit and a first RMSNorm normalization computing subunit; the multi-cluster attention module includes a second configurable normalization computing unit, a feature extraction computing unit, a multi-cluster attention mechanism computing unit, an output linear mapping computing unit and a first configurable residual link computing unit; wherein the second configurable normalization computing unit includes: a second non-normalization subunit, a second LayerNorm normalization computing subunit and a second RMSNorm normalization computing subunit; the first configurable residual link computing unit includes a first non-residual link computing subunit and a first residual link computing subunit; the multi-layer perceptron module includes a third configurable normalization computing unit, a linear dimension raising computing unit, a configurable activation function computing unit, a linear The invention relates to a method for realizing a multi-cluster attention mechanism computing unit, a multi-cluster attention mechanism computing unit and a multi-cluster attention mechanism computing unit, wherein the multi-cluster attention mechanism computing unit comprises a first non-normalization subunit, a second LayerNorm normalization subunit and a third RMSNorm normalization subunit; the multi-cluster attention mechanism computing unit comprises a first non-normalization subunit, a second LayerNorm normalization subunit and a third RMSNorm normalization subunit; the multi-cluster attention mechanism computing unit comprises a first non-normalization subunit, a second LayerNorm normalization subunit and a third RMSNorm normalization subunit; the multi-cluster attention mechanism computing unit comprises a first non-normalization subunit, a second LayerNorm normalization subunit and a third RMSNorm normalization subunit; the multi-cluster attention mechanism computing unit comprises a second ... first non-normalization subunit, a second LayerNorm normalization subunit and a third RMSNorm normalization subunit; the multi-cluster attention mechanism computing unit comprises a second non-normalization subunit, a second LayerNorm normalization subunit and a third R It solves the problems in the existing technology that are limited by the external storage bandwidth, making it difficult to fully utilize the computing efficiency of the chip, the poor performance and the high requirements for professionals. It realizes the construction of a more complete large language model and designs a high-throughput computing data flow, which can fully utilize the computing power of the chip.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 2 is a schematic diagram of the structure of a large language model modular reasoning computing system provided according to the first embodiment of the present invention;

[0029] Figure 2 is a flowchart of a large language model modular reasoning calculation method provided according to the second embodiment of the present invention;

[0030] Figure 3 2 is a schematic diagram of the structure of a large language model modular reasoning computing device provided according to Embodiment 3 of the present invention;

[0031] Figure 4 It is a schematic diagram of the structure of an electronic device provided according to Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "target", "current", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment 1

[0035] Figure 11 is a schematic diagram of a large language model modular reasoning computing system according to Embodiment 1 of the present invention. The large language model modular reasoning computing system 100 includes a text embedding module 110, a multi-cluster attention module 120, a multi-layer perceptron module 130 and a linear mapping module 140.

[0036] Wherein, the text embedding module 110 includes a first configurable normalization calculation unit; the first configurable normalization calculation unit includes: a first non-normalization subunit, a first LayerNorm normalization calculation subunit and a first RMSNorm normalization calculation subunit;

[0037] The multi-cluster attention module 120 includes a second configurable normalization calculation unit, a feature extraction calculation unit, a multi-cluster attention mechanism calculation unit, an output linear mapping calculation unit and a first configurable residual link calculation unit; wherein the second configurable normalization calculation unit includes: a second non-normalization subunit, a second LayerNorm normalization calculation subunit and a second RMSNorm normalization calculation subunit; the first configurable residual link calculation unit includes a first non-residual link calculation subunit and a first residual link calculation subunit;

[0038] The multi-layer perceptron module 130 includes a third configurable normalization calculation unit, a linear dimension increase calculation unit, a configurable activation function calculation unit, a linear dimension reduction calculation unit and a second configurable residual link calculation unit; wherein the third configurable normalization calculation unit includes: a third non-normalization subunit, a third LayerNorm normalization calculation subunit and a third RMSNorm normalization calculation subunit; the configurable activation function calculation unit includes a calculation unit of at least one activation function type; the second configurable residual link calculation unit includes a second non-residual link calculation subunit and a second residual link calculation subunit;

[0039] The linear mapping module 140 includes a fourth configurable normalization calculation unit and a linear probability mapping calculation unit; wherein the fourth configurable normalization calculation unit includes: a fourth non-normalization subunit, a fourth LayerNorm normalization calculation subunit and a fourth RMSNorm normalization calculation subunit;

[0040] Wherein, the multi-cluster attention mechanism calculation unit is calculated based on an improved attention mechanism calculation data flow method;

[0041] The feature extraction calculation unit, the output linear mapping calculation unit, the feature extraction calculation unit, the output linear mapping calculation unit and the linear probability mapping calculation unit are all calculated based on an improved linear layer calculation data flow method.

[0042] In this embodiment, a large language model modular reasoning computing system is modularly designed, which may include a text embedding module, a multi-cluster attention module, a multi-layer perceptron module, and a linear mapping module.

[0043] For the text embedding module, you can choose not to perform normalization or to perform normalization. For normalization, you can use LayerNorm or RMSNorm to perform the normalization operation.

[0044] In detail, the text embedding module may include a first configurable normalization calculation unit. The first configurable normalization calculation unit may include a first non-normalization subunit, a first LayerNorm normalization calculation subunit, and a first RMSNorm normalization calculation subunit.

[0045] For the multi-cluster attention module, you can choose not to perform normalization or to perform normalization. The calculation method is the same as that of the text embedding module, which will not be repeated here.

[0046] In addition, the multi-cluster attention module can also include a feature extraction calculation unit, which can perform linear mapping QKV feature extraction to obtain Q linear mapping feature extraction results, K linear mapping feature extraction results and V linear mapping feature extraction results, where Q is the query feature matrix; K is the key feature matrix; V is the value feature matrix.

[0047] Among them, the multi-cluster attention module can also include a multi-cluster attention mechanism calculation unit, which is calculated based on an improved attention mechanism calculation data flow method. The output linear mapping calculation unit is calculated based on an improved linear layer calculation data flow method. The multi-cluster attention module can also include a first configurable residual link calculation unit; wherein the first configurable residual link calculation unit can include a first non-residual link calculation subunit and a first residual link calculation subunit. That is, for the first configurable residual link calculation unit, it can be selected not to perform residual link calculation or to perform residual link calculation.

[0048] In addition, for the multi-layer perceptron module, you can choose not to perform normalization or to perform normalization. The calculation method is the same as that of the text embedding module, so I will not go into details here.

[0049] Specifically, the multi-layer perceptron module may include a linear dimension increase calculation unit, a configurable activation function calculation unit, a linear dimension reduction calculation unit and a second configurable residual link calculation unit.

[0050] In detail, the configurable activation function calculation unit includes a calculation unit of at least one activation function type; the activation function type may be ReLU, GeLU, SiLU or SwiGLU, etc. In addition, for the second configurable residual link calculation unit, it is possible to choose not to perform residual link calculation or to perform residual link calculation.

[0051] In this embodiment, for the linear mapping module, it is also possible to choose not to perform normalization processing or to perform normalization processing, which is the same as the calculation method of the text embedding module and will not be described in detail here.

[0052] In addition, the linear mapping module also includes a linear probability mapping calculation unit.

[0053] For the feature extraction calculation unit, the output linear mapping calculation unit, the feature extraction calculation unit, the output linear mapping calculation unit and the linear probability mapping calculation unit, the calculations are all performed based on the improved linear layer calculation data flow method.

[0054] In this embodiment, a large language model network can be quickly built using a modular construction method. Users do not need to be aware of the implementation details of each module, and the model can be quickly built and deployed, reducing the requirements for professionals.

[0055] The technical solution of the embodiment of the present invention, the large language model modular reasoning computing system includes a text embedding module, a multi-cluster attention module, a multi-layer perceptron module and a linear mapping module; wherein the text embedding module includes a first configurable normalization computing unit; the first configurable normalization computing unit includes: a first non-normalization subunit, a first LayerNorm normalization computing subunit and a first RMSNorm normalization computing subunit; the multi-cluster attention module includes a second configurable normalization computing unit, a feature extraction computing unit, a multi-cluster attention mechanism computing unit, an output linear mapping computing unit and a first configurable residual link computing unit; wherein the second configurable normalization computing unit includes: a second non-normalization subunit, a second LayerNorm normalization computing subunit and a second RMSNorm normalization computing subunit; the first configurable residual link computing unit includes a first non-residual link computing subunit and a first residual link computing subunit; the multi-layer perceptron module includes a third configurable normalization computing unit, a linear dimension raising computing unit, a configurable activation function computing unit, a linear The invention relates to a method for realizing a multi-cluster attention mechanism computing unit, a multi-cluster attention mechanism computing unit and a multi-cluster attention mechanism computing unit, wherein the multi-cluster attention mechanism computing unit comprises a first non-normalization subunit, a second LayerNorm normalization subunit and a third RMSNorm normalization subunit; the multi-cluster attention mechanism computing unit comprises a first non-normalization subunit, a second LayerNorm normalization subunit and a third RMSNorm normalization subunit; the multi-cluster attention mechanism computing unit comprises a first non-normalization subunit, a second LayerNorm normalization subunit and a third RMSNorm normalization subunit; the multi-cluster attention mechanism computing unit comprises a first non-normalization subunit, a second LayerNorm normalization subunit and a third RMSNorm normalization subunit; the multi-cluster attention mechanism computing unit comprises a second ... first non-normalization subunit, a second LayerNorm normalization subunit and a third RMSNorm normalization subunit; the multi-cluster attention mechanism computing unit comprises a second non-normalization subunit, a second LayerNorm normalization subunit and a third R It solves the problems in the existing technology that are limited by the external storage bandwidth, making it difficult to fully utilize the computing efficiency of the chip, the poor performance and the high requirements for professionals. It realizes the construction of a more complete large language model and designs a high-throughput computing data flow, which can fully utilize the computing power of the chip.

[0056] Embodiment 2

[0057] Figure 2A flowchart of a large language model modular reasoning calculation method is provided for the second embodiment of the present invention. This embodiment is applicable to the modular reasoning calculation corresponding to the large language model. The method can be executed by a large language model modular reasoning calculation device, specifically, by a large language model modular reasoning calculation system. The large language model modular reasoning calculation device can be implemented in the form of hardware and / or software.

[0058] Correspondingly, such as Figure 2 As shown, the method includes:

[0059] S210, acquiring an input text token sequence to be calculated in real time, performing feature vector search processing on the input text token sequence to be calculated through a text embedding module, and performing configurable normalization processing through a first configurable normalization calculation unit to obtain a feature vector of the input text token sequence.

[0060] The input text token sequence to be calculated may be a sequence of input text tokens that need to be calculated.

[0061] Exemplarily, the input text token sequence to be calculated can be set to , n represents the total length of the input text token sequence to be calculated; i represents the input text token; the value range of t is [0, n-1].

[0062] Optionally, the real-time acquisition of the input text token sequence to be calculated, the feature vector search processing of the input text token sequence to be calculated by the text embedding module, and the configurable normalization processing by the first configurable normalization calculation unit to obtain the input text token sequence feature vector, including: real-time acquisition of the input text token sequence to be calculated, and acquisition of the target token lookup table dictionary corresponding to the text embedding module; wherein the target token lookup table dictionary is , N represents the total length of the target token lookup table dictionary, and h represents the feature vector dimension; in the target token lookup table dictionary, feature vector search processing is performed according to each input text token in the input text token sequence to obtain the input text token sequence search feature vector sequence; wherein, the input text token sequence search feature vector sequence is , Represents an input text token sequence search feature vector; if the first configurable normalization calculation unit is the first non-normalization subunit, each input text token sequence search feature vector is determined as an input text token sequence feature vector; if the first configurable normalization calculation unit is the first LayerNorm normalization calculation subunit, the input text token sequence search feature vector sequence is normalized by the LayerNorm normalization calculation formula to obtain the input text token sequence feature vector y; wherein the LayerNorm normalization calculation formula is ; is the external scaling factor, is the external offset factor, is a constant; if the first configurable normalization calculation unit is the first RMSNorm normalization calculation subunit, the input text token sequence is searched for the feature vector sequence and normalized by the RMSNorm normalization calculation formula to obtain the input text token sequence feature vector y; wherein the RMSNorm normalization calculation formula is ;in, .

[0063] In this embodiment, firstly, feature vector search processing is performed according to each input text token in the input text token sequence, that is, In the , , The input text token is used to perform a feature vector lookup process in the target token lookup table dictionary.

[0064] Assume that The input text token sequence to find the feature vector is ;pass The input text token sequence to find the feature vector is ;pass The input text token sequence to find the feature vector is ; By analogy, we can get the input text token sequence to find the feature vector sequence: .

[0065] Further, in getting Afterwards, you can choose not to perform normalization, LayerNorm normalization, or RMSNorm normalization according to the pre-configured conditions.

[0066] For LayerNorm normalization, , normalize the input text token sequence to find the feature vector sequence, and obtain the input text token sequence feature vector. For selecting RMSNorm normalization, by , normalize the input text token sequence to find the feature vector sequence, and get the input text token sequence feature vector, where, .

[0067] S220, inputting the input text token sequence feature vector into the multi-layer loop corresponding to the large language model, and looping between the multi-cluster attention module and the multi-layer perceptron module layer by layer until all layers of loops are traversed to obtain the linear calculation result of the input text token sequence.

[0068] In this embodiment, assuming a 3-layer loop corresponding to a large language model, it is necessary to first input the feature vector of the input text token sequence into the multi-cluster attention module. After the multi-cluster attention module completes the processing, the result is sent to the multi-layer perceptron module for processing. After the processing result is obtained, it is determined that the first layer loop operation is completed.

[0069] Then the first layer loop result is returned to the multi-cluster attention module for processing, and then looped between the multi-cluster attention module and the multi-layer perceptron module in turn until the three-layer loop operation is completed, and the linear calculation result of the input text token sequence can be obtained.

[0070] Optionally, it also includes: in the feature extraction calculation unit corresponding to the multi-cluster attention module, linear mapping QKV feature extraction is performed on the input text token sequence feature vector to obtain Q linear mapping feature extraction results, K linear mapping feature extraction results and V linear mapping feature extraction results respectively; wherein the Q linear mapping feature extraction result is ; The K linear mapping feature extraction result is ; The linear mapping feature extraction result of V is ; , and They are the linear layer weight tensors of QKV respectively; , and They are the linear layer bias tensors of QKV respectively; Q is the query feature matrix; K is the key feature matrix; V is the value feature matrix.

[0071] In addition, for the K linear mapping feature extraction results and the V linear mapping feature extraction results, position coding calculations may be performed according to pre-configured parameters, including not performing position coding calculations or performing position coding calculations. Among them, the position coding calculations may include 2D rotation coding or 1D rotation coding.

[0072] In addition, the large language model performs modular reasoning calculations based on an artificial intelligence dedicated acceleration chip; the storage architecture of the artificial intelligence dedicated acceleration chip includes on-chip private storage, on-chip shared storage and external storage; wherein the space size of the external storage is larger than the on-chip shared storage, and the space size of the on-chip shared storage is larger than the on-chip private storage.

[0073] Exemplarily, it is assumed that the external storage is L3 storage, the on-chip shared storage is L2 storage, and the on-chip private storage is L1 storage. Specifically, the space size of L3 storage is larger than that of L2 storage, and the space size of L2 storage is larger than that of L1 storage.

[0074] Specifically, for AI-specific acceleration chips, L2 storage can be accessed by each computing core inside the chip. L1 storage can only be accessed by the computing core to which it belongs, and cannot be accessed by other computing cores. Each computing core inside the chip contains vector computing cores and tensor computing cores as the main computing units, and also supports scalar computing, etc.

[0075] The tensor calculation core is used for matrix multiplication calculation, which mainly realizes the improved attention mechanism calculation data flow method.

[0076] The vector computing core is used for multi-data parallel computing. For example, the data width is 1024 bits, and it supports 32 floating-point numbers or 64 half-precision floating-point numbers in parallel. It is used to perform calculations other than matrix operations, including element-by-element addition, subtraction, multiplication and division, transcendental function Softmax, or activation functions (such as ReLU, GeLU or SwiGLU, etc.).

[0077] In the process of constructing a large language model, the improved attention mechanism calculation data flow method and the improved linear layer calculation data flow method have a crucial impact on the performance of the model. According to the architecture of the AI ​​dedicated acceleration chip, a proprietary data flow can be designed for the above two calculation modes to give full play to the computing power of the chip.

[0078] Specifically, in the feature extraction calculation unit corresponding to the multi-cluster attention module, after performing linear mapping QKV feature extraction on the input text token sequence feature vector to obtain the Q linear mapping feature extraction result, the K linear mapping feature extraction result and the V linear mapping feature extraction result respectively, it also includes: in the input text token sequence search feature vector sequence, sequentially obtaining an input text token sequence feature vector in sequence; performing matrix sub-block segmentation on the first input text token sequence feature vector to obtain the matrix sub-block segmentation result of each first input text token sequence feature vector; respectively performing linear layer weight tensors in the Q linear mapping feature extraction result, the K linear mapping feature extraction result and the V linear mapping feature extraction result , and The matrix sub-blocks are divided respectively, and according to the matrix sub-block division results of each first input text token sequence feature vector, the data flow arrangement processing is performed through the improved linear layer calculation data flow method, and the data flow is transferred between the on-chip private storage, the on-chip shared storage and the external storage, and all the matrix sub-block division results of the first input text token sequence feature vector and all the linear layer weight tensor matrix sub-block division results are traversed to obtain the data flow arrangement result of the first input text token sequence feature vector; return to execute the operation of sequentially obtaining an input text token sequence feature vector in the input text token sequence search feature vector sequence, until the traversal of each input text token sequence feature vector is completed.

[0079] For example, the linear mapping feature extraction result for Q For example, we can first divide the first input text token sequence feature vector into matrix sub-blocks, that is, Perform matrix sub-block segmentation. Assuming it is segmented into m blocks, we can get , you can The matrix is ​​also divided into sub-blocks. Assuming that it can be divided into m blocks, we can get .for , and For example, initially, they are all stored on L3 storage, and the size of each matrix sub-block is the same. Two linear layer weight tensor matrix sub-blocks are prepared on L2 storage and L1 storage as ping-pong cache for parallel data transmission and calculation, and one input feature matrix sub-block and one linear layer bias tensor matrix are prepared on L1 storage.

[0080] The specific steps are as follows:

[0081] a) Read The corresponding first feature matrix sub-block , because for , and For example, initially they are all stored on L3 storage. From L3 storage directly to L1 storage.

[0082] b) Read The corresponding first linear layer weight tensor matrix sub-block ,Will From L3 storage to the first ping-pong cache of L2 storage; read the complete The linear layer bias tensor matrix is From L3 storage directly to L1 storage.

[0083] c) Read The corresponding next linear layer weight tensor matrix sub-block ,Will From L3 storage to the second ping-pong cache in L2 storage; read the previous storage in the first ping-pong cache in L2 storage The corresponding first linear layer weight tensor matrix sub-block , and From L2 storage to the first ping-pong cache of L1 storage.

[0084] d) Continue reading The corresponding next linear layer weight tensor matrix sub-block ,Will From L3 storage to the first ping-pong cache in L2 storage; read the previous storage in the second ping-pong cache in L2 storage , and From L2 storage to the second ping-pong cache of L1 storage; and Perform linear matrix multiplication and save the calculated result Q11 in L1 storage.

[0085] e) Continue reading The corresponding next linear layer weight tensor matrix sub-block ,Will From L3 storage to the second ping-pong cache of L2 storage; read the previous storage in the first ping-pong cache of L2 storage , and From L2 storage to the first ping-pong cache of L1 storage. Then and Perform linear matrix multiplication calculation, save the calculated result Q12 in L1 storage, and transfer the calculation result Q11 of the previous step from L1 storage to L2 storage for storage.

[0086] f) Continue to follow the above steps until all feature matrix sub-blocks and linear layer weight tensor matrix sub-blocks are traversed, and finally the input text token sequence feature vector data stream arrangement result is obtained.

[0087] Similarly, the K linear mapping feature extraction results can be ; V linear mapping feature extraction results The corresponding processing is performed and will not be elaborated here.

[0088] The advantage of this setting is that it can achieve complete parallelism of data transmission and calculation, greatly improve data throughput, and make full use of the chip's bandwidth and computing power.

[0089] Optionally, it also includes: in the multi-cluster attention mechanism calculation unit corresponding to the multi-cluster attention module, obtaining the Q linear mapping feature extraction result, the K linear mapping feature extraction result and the V linear mapping feature extraction result; calculating the data flow method and formula through the improved attention mechanism , and get the calculation results of QKV multi-cluster attention mechanism ; Among them, scale is the external scaling factor; mask is the mask; Softmax is the transcendental function; mask corresponds to the mask matrix.

[0090] Among them, the method and formula for calculating data flow through the improved attention mechanism , and obtain the calculation results of the QKV multi-cluster attention mechanism, including: performing matrix sub-block segmentation on the query feature matrix, key feature matrix, value feature matrix, and mask matrix, and obtaining the sub-block segmentation results of each query feature matrix, each key feature matrix, each value feature matrix, and each mask matrix; the improved attention mechanism is used to calculate the data flow method and formula , the query feature matrix sub-block segmentation results, the key feature matrix sub-block segmentation results, the value feature matrix sub-block segmentation results and the mask matrix sub-block segmentation results are arranged and calculated in data streams, and data is transferred between on-chip private storage, on-chip shared storage and external storage until each query feature matrix sub-block segmentation result traverses each key feature matrix sub-block segmentation result and each value feature matrix sub-block segmentation result to obtain the calculation result of the QKV multi-cluster attention mechanism.

[0091] In this embodiment, the query feature matrix Q, the key feature matrix K, the value feature matrix V, and the mask matrix Perform matrix sub-block splitting, assuming ; ; ; ; The sizes of the sub-block matrices cut out by K and V are equal. The shape of the mask matrix and The shapes of the matrix calculation results of the sub-blocks are the same.

[0092] Specifically, the logic of FlashAttention is applied to the calculation of the attention mechanism to support the calculation of very long sequences, using a loop iteration method. Here, the calculation results of each matrix sub-block need to be saved in the L2 storage for the next matrix sub-block calculation to be used for iterative update. Different from the improved linear layer calculation data flow method, which uses a double ping-pong cache method to perform calculations and data transmission in parallel, the improved attention mechanism calculation data flow method contains more diverse and complex calculation types.

[0093] The specific calculation steps are as follows:

[0094] a) Read the first Q query feature sub-matrix ,Will From L3 storage to L2 storage.

[0095] b) Read the first K-key feature submatrix ,Will From L3 storage to L2 storage; read the first Q query feature submatrix ,Will From L2 storage to L1 storage; read the corresponding Mask mask sub-matrix , directly store it from L3 into L1 storage.

[0096] c) Read the first V value feature sub-matrix ,Will From L3 storage to L2 storage; read the first K-key feature submatrix , from L2 storage to L1 storage.

[0097] d) Read the first V value feature sub-matrix , from L2 storage to L1 storage; calculate the first Q query feature submatrix with the first K key eigenmatrices The matrix product of the transpose of .

[0098] e) Based on the calculation result of d, calculate .

[0099] f) According to the calculation result of e, do Softmax and get the output result .

[0100] g) The calculation results of f and the V-value characteristic submatrix Do matrix multiplication, that is ; If this is not the first cycle, read the previous step Output the results from L2 storage to L1 storage, ready for loop iterative update.

[0101] h) Read the next Mask sub-matrix , directly from L3 storage to L1 storage; read the next K-key feature submatrix , from L3 storage to L2 storage; the calculation result of g is written back from L1 storage to L2 storage for caching, and is used for subsequent FlashAttention loop iteration calculations; if the current cycle is not the first round, the element-by-element calculation in FlashAttention is executed to iteratively update .

[0102] i) Continue to execute the above steps until all K key feature submatrices and V value feature submatrices are traversed for each Q query feature submatrix. The final result O comes from the concatenation of the attention calculation results of all K and V feature submatrices traversed for each Q query feature submatrix.

[0103] The advantage of this setting is that by interleaving different types of calculations and corresponding data transmissions to achieve parallel pipelining, the computing power of the chip can be fully utilized to improve system performance.

[0104] S230, inputting the linear calculation result of the input text token sequence into the linear mapping module for token probability mapping processing to obtain the generated word tokens of each input text.

[0105] In this embodiment, the linear calculation result of the input text token sequence is linearly mapped to output dimension information consistent with the dictionary length; wherein the target token lookup table dictionary is , N represents the total length of the target token lookup table dictionary. Further, softmax calculation is performed on the vector mapped by the linear layer, and the output of the linear layer is converted into a probability value to obtain the generated word token of the current input text, and then the generated word token of each input text can be obtained.

[0106] S240, generating word tokens from each input text and providing feedback to the user.

[0107] The technical solution of the embodiment of the present invention obtains the input text token sequence to be calculated in real time, performs feature vector search processing on the input text token sequence to be calculated through the text embedding module, and performs configurable normalization processing through the first configurable normalization calculation unit to obtain the input text token sequence feature vector; the input text token sequence feature vector is input into the multi-layer loop corresponding to the large language model, and is looped between the multi-cluster attention module and the multi-layer perceptron module layer by layer until each layer of loop is traversed to obtain the linear calculation result of the input text token sequence; the linear calculation result of the input text token sequence is input into the linear mapping module for token probability mapping processing to obtain each input text generated word token; each input text generated word token is fed back to the user. The problem that the chip's computing efficiency is difficult to fully utilize, the performance is poor, and the requirements for professionals are high in the prior art due to the limitation of external storage bandwidth is solved, and a more complete large language model is built, and a high-throughput computing data flow is designed to fully utilize the computing power of the chip.

[0108] Embodiment 3

[0109] Figure 3 This is a schematic diagram of the structure of a large language model modular reasoning computing device provided in the third embodiment of the present invention. The large language model modular reasoning computing device provided in this embodiment can be implemented by software and / or hardware, and can be configured in a terminal device or a server to implement a large language model modular reasoning computing method in the embodiment of the present invention. Figure 3 As shown, the device includes: an input text token sequence feature vector determination module 310, an input text token sequence linear calculation result determination module 320, each input text generated word token determination module 330 and each input text generated word token feedback module 340.

[0110] The input text token sequence feature vector determination module 310 is used to obtain the input text token sequence to be calculated in real time, perform feature vector search processing on the input text token sequence to be calculated through the text embedding module, and perform configurable normalization processing through the first configurable normalization calculation unit to obtain the input text token sequence feature vector;

[0111] The input text token sequence linear calculation result determination module 320 is used to input the input text token sequence feature vector into the multi-layer loop corresponding to the large language model, and loop between the multi-cluster attention module and the multi-layer perceptron module layer by layer until each layer of loop is traversed to obtain the input text token sequence linear calculation result;

[0112] Each input text generated word token determination module 330 is used to input the linear calculation result of the input text token sequence into the linear mapping module for token probability mapping processing to obtain each input text generated word token;

[0113] The input text generated word token feedback module 340 is used to feed back the word token generated by each input text to the user.

[0114] The technical solution of the embodiment of the present invention obtains the input text token sequence to be calculated in real time, performs feature vector search processing on the input text token sequence to be calculated through the text embedding module, and performs configurable normalization processing through the first configurable normalization calculation unit to obtain the input text token sequence feature vector; the input text token sequence feature vector is input into the multi-layer loop corresponding to the large language model, and is looped between the multi-cluster attention module and the multi-layer perceptron module layer by layer until each layer of loop is traversed to obtain the linear calculation result of the input text token sequence; the linear calculation result of the input text token sequence is input into the linear mapping module for token probability mapping processing to obtain each input text generated word token; each input text generated word token is fed back to the user. The problem that the chip's computing efficiency is difficult to fully utilize, the performance is poor, and the requirements for professionals are high in the prior art due to the limitation of external storage bandwidth is solved, and a more complete large language model is built, and a high-throughput computing data flow is designed to fully utilize the computing power of the chip.

[0115] Based on the above embodiments, the input text token sequence feature vector determination module 310 can be specifically used to: obtain the input text token sequence to be calculated in real time, and obtain the target token lookup table dictionary corresponding to the text embedding module; wherein the input text token sequence to be calculated is , n represents the total length of the input text token sequence to be calculated; i represents the input text token; the value range of t is [0, n-1]; the target token lookup table dictionary is , N represents the total length of the target token lookup table dictionary, and h represents the feature vector dimension; in the target token lookup table dictionary, feature vector search processing is performed according to each input text token in the input text token sequence to obtain the input text token sequence search feature vector sequence; wherein, the input text token sequence search feature vector sequence is , Represents an input text token sequence search feature vector; if the first configurable normalization calculation unit is the first non-normalization subunit, each input text token sequence search feature vector is determined as an input text token sequence feature vector; if the first configurable normalization calculation unit is the first LayerNorm normalization calculation subunit, the input text token sequence search feature vector sequence is normalized by the LayerNorm normalization calculation formula to obtain the input text token sequence feature vector y; wherein the LayerNorm normalization calculation formula is ; is the external scaling factor, is the external offset factor, is a constant; if the first configurable normalization calculation unit is the first RMSNorm normalization calculation subunit, the input text token sequence is searched for the feature vector sequence and normalized by the RMSNorm normalization calculation formula to obtain the input text token sequence feature vector y; wherein the RMSNorm normalization calculation formula is ;in, .

[0116] On the basis of the above embodiments, it also includes a QKV linear mapping feature extraction result determination result, which can be specifically used for: in the feature extraction calculation unit corresponding to the multi-cluster attention module, linear mapping QKV feature extraction is performed on the input text token sequence feature vector to obtain a Q linear mapping feature extraction result, a K linear mapping feature extraction result and a V linear mapping feature extraction result respectively; wherein the Q linear mapping feature extraction result is ; The K linear mapping feature extraction result is ; The linear mapping feature extraction result of V is ; , and They are the linear layer weight tensors of QKV respectively; , and They are the linear layer bias tensors of QKV respectively; Q is the query feature matrix; K is the key feature matrix; V is the value feature matrix.

[0117] On the basis of the above embodiments, it also includes a QKV multi-cluster attention mechanism calculation result determination module, which can be specifically used to: obtain the Q linear mapping feature extraction result, the K linear mapping feature extraction result and the V linear mapping feature extraction result in the multi-cluster attention mechanism calculation unit corresponding to the multi-cluster attention module; calculate the data flow method and formula through the improved attention mechanism , and get the calculation results of QKV multi-cluster attention mechanism ; Among them, scale is the external scaling factor; mask is the mask; Softmax is the transcendental function; mask corresponds to the mask matrix.

[0118] Based on the above embodiments, the large language model performs modular reasoning calculations based on an artificial intelligence dedicated acceleration chip; the storage architecture of the artificial intelligence dedicated acceleration chip includes on-chip private storage, on-chip shared storage and external storage; wherein the space size of the external storage is larger than the on-chip shared storage, and the space size of the on-chip shared storage is larger than the on-chip private storage.

[0119] On the basis of the above embodiments, it also includes an input text token sequence feature vector data stream arrangement result determination module, which can be specifically used for: in the feature extraction calculation unit corresponding to the multi-cluster attention module, linear mapping QKV feature extraction is performed on the input text token sequence feature vector, and after obtaining the Q linear mapping feature extraction result, the K linear mapping feature extraction result and the V linear mapping feature extraction result respectively, in the input text token sequence search feature vector sequence, one input text token sequence feature vector is obtained in sequence; the first input text token sequence feature vector is matrix sub-block segmented to obtain the matrix sub-block segmentation result of each first input text token sequence feature vector; the linear layer weight tensor in the Q linear mapping feature extraction result, the K linear mapping feature extraction result and the V linear mapping feature extraction result is respectively , and Perform matrix sub-block segmentation, and perform data stream arrangement processing through an improved linear layer data stream calculation method according to the matrix sub-block segmentation results of each first input text token sequence feature vector, and transfer data between on-chip private storage, on-chip shared storage and external storage, traverse all the matrix sub-block segmentation results of the first input text token sequence feature vector and all the linear layer weight tensor matrix sub-block segmentation results to obtain the data stream arrangement result of the first input text token sequence feature vector; return to execute the operation of sequentially obtaining an input text token sequence feature vector in the input text token sequence search feature vector sequence, until the traversal of each input text token sequence feature vector is completed.

[0120] On the basis of the above embodiments, the QKV multi-cluster attention mechanism calculation result determination module can also be specifically used to: respectively perform matrix sub-block segmentation on the query feature matrix, the key feature matrix, the value feature matrix, and the mask matrix, and obtain the sub-block segmentation results of each query feature matrix, each key feature matrix sub-block segmentation result, each value feature matrix sub-block segmentation result, and each mask matrix sub-block segmentation result; calculate the data flow method and formula through the improved attention mechanism , the query feature matrix sub-block segmentation results, the key feature matrix sub-block segmentation results, the value feature matrix sub-block segmentation results and the mask matrix sub-block segmentation results are arranged and calculated in data streams, and data is transferred between on-chip private storage, on-chip shared storage and external storage until each query feature matrix sub-block segmentation result traverses each key feature matrix sub-block segmentation result and each value feature matrix sub-block segmentation result to obtain the calculation result of the QKV multi-cluster attention mechanism.

[0121] The large language model modular reasoning and computing device provided in the embodiment of the present invention can execute the large language model modular reasoning and computing method provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.

[0122] Embodiment 4

[0123] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement the fourth embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0124] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0125] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0126] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a large language model modular reasoning computing method.

[0127] In some embodiments, the large language model modular reasoning calculation method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the large language model modular reasoning calculation method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the large language model modular reasoning calculation method in any other appropriate manner (for example, by means of firmware).

[0128] The method comprises: obtaining an input text token sequence to be calculated in real time, performing feature vector search processing on the input text token sequence to be calculated through a text embedding module, and performing configurable normalization processing through a first configurable normalization calculation unit to obtain an input text token sequence feature vector; inputting the input text token sequence feature vector into a multi-layer loop corresponding to a large language model, performing loop processing between a multi-cluster attention module and a multi-layer perceptron module layer by layer, until each layer of loop is traversed and completed, to obtain an input text token sequence linear calculation result; inputting the input text token sequence linear calculation result into the linear mapping module to perform token probability mapping processing, to obtain each input text generated word token; and feeding back each input text generated word token to a user.

[0129] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0130] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0131] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0133] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0134] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0135] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0136] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0137] Embodiment 5

[0138] Embodiment 5 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions are used to execute a large language model modular reasoning calculation method when executed by a computer processor, the method comprising: obtaining an input text token sequence to be calculated in real time, performing feature vector search processing on the input text token sequence to be calculated through a text embedding module, and performing configurable normalization processing through a first configurable normalization calculation unit to obtain an input text token sequence feature vector; inputting the input text token sequence feature vector into a multi-layer loop corresponding to the large language model, looping between a multi-cluster attention module and a multi-layer perceptron module layer by layer, until each layer of the loop is traversed and completed, to obtain an input text token sequence linear calculation result; inputting the input text token sequence linear calculation result into the linear mapping module for token probability mapping processing to obtain each input text generated word token; and feeding back each input text generated word token to a user.

[0139] Of course, the computer-readable storage medium provided in an embodiment of the present invention has computer executable instructions that are not limited to the method operations described above, and can also execute related operations in the modular reasoning calculation of the large language model provided in any embodiment of the present invention.

[0140] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0141] It is worth noting that in the above-mentioned embodiment of modular reasoning calculation of large language model, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0142] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A large language model modular reasoning computing system, characterized in that: The large language model modular reasoning computing system includes a text embedding module, a multi-cluster attention module, a multi-layer perceptron module and a linear mapping module; Wherein, the text embedding module includes a first configurable normalization calculation unit; the first configurable normalization calculation unit includes: a first non-normalization subunit, a first LayerNorm normalization calculation subunit and a first RMSNorm normalization calculation subunit; The multi-cluster attention module includes a second configurable normalization calculation unit, a feature extraction calculation unit, a multi-cluster attention mechanism calculation unit, an output linear mapping calculation unit and a first configurable residual link calculation unit; wherein the second configurable normalization calculation unit includes: a second non-normalization subunit, a second LayerNorm normalization calculation subunit and a second RMSNorm normalization calculation subunit; the first configurable residual link calculation unit includes a first non-residual link calculation subunit and a first residual link calculation subunit; The multi-layer perceptron module includes a third configurable normalization calculation unit, a linear dimension increase calculation unit, a configurable activation function calculation unit, a linear dimension reduction calculation unit and a second configurable residual link calculation unit; wherein the third configurable normalization calculation unit includes: a third non-normalization subunit, a third LayerNorm normalization calculation subunit and a third RMSNorm normalization calculation subunit; the configurable activation function calculation unit includes a calculation unit of at least one activation function type; the second configurable residual link calculation unit includes a second non-residual link calculation subunit and a second residual link calculation subunit; The linear mapping module includes a fourth configurable normalization calculation unit and a linear probability mapping calculation unit; wherein the fourth configurable normalization calculation unit includes: a fourth non-normalization subunit, a fourth LayerNorm normalization calculation subunit and a fourth RMSNorm normalization calculation subunit; Wherein, the multi-cluster attention mechanism calculation unit is calculated based on an improved attention mechanism calculation data flow method; The feature extraction calculation unit, the output linear mapping calculation unit and the linear probability mapping calculation unit are all calculated based on an improved linear layer calculation data flow method.

2. A large language model modular reasoning calculation method, characterized in that: It is executed by a large language model modular reasoning computing system, including: Obtaining in real time the input text token sequence to be calculated, and obtaining the target token lookup table dictionary corresponding to the text embedding module; Among them, the input text token sequence to be calculated is , n represents the total length of the input text token sequence to be calculated; i represents the input text token; the value range of t is [0, n-1]; the target token lookup table dictionary is , N represents the total length of the target token lookup table dictionary, and h represents the feature vector dimension; In the target token lookup table dictionary, feature vector lookup processing is performed according to each input text token in the input text token sequence to obtain an input text token sequence lookup feature vector sequence; Among them, the input text token sequence to find the feature vector sequence is , Represents the input text token sequence to find the feature vector; If the first configurable normalization calculation unit is a first non-normalization subunit, each input text token sequence search feature vector is determined as the input text token sequence feature vector; If the first configurable normalization calculation unit is a first LayerNorm normalization calculation subunit, the input text token sequence is normalized by searching the feature vector sequence using the LayerNorm normalization calculation formula to obtain the input text token sequence feature vector y; Among them, the LayerNorm normalization calculation formula is: ; is the external scaling factor, is the external offset factor, is a constant; If the first configurable normalization calculation unit is a first RMSNorm normalization calculation subunit, the input text token sequence is normalized by searching the feature vector sequence using the RMSNorm normalization calculation formula to obtain the input text token sequence feature vector y; Among them, the RMSNorm normalization calculation formula is: ;in, ; Input the input text token sequence feature vector into the multi-layer loop corresponding to the large language model, and loop through the multi-cluster attention module and the multi-layer perceptron module layer by layer until all layers of loops are traversed to obtain the linear calculation result of the input text token sequence; Inputting the linear calculation result of the input text token sequence into the linear mapping module for token probability mapping processing to obtain the generated word tokens of each input text; The word tokens generated from each input text are fed back to the user.

3. The method according to claim 2, characterized in that Also includes: In the feature extraction calculation unit corresponding to the multi-cluster attention module, linear mapping QKV feature extraction is performed on the input text token sequence feature vector to obtain Q linear mapping feature extraction results, K linear mapping feature extraction results and V linear mapping feature extraction results respectively; Among them, the Q linear mapping feature extraction result is ; The K linear mapping feature extraction result is ; The linear mapping feature extraction result of V is ; , and They are the linear layer weight tensors of QKV respectively; , and They are the linear layer bias tensors of QKV respectively; Q is the query feature matrix; K is the key feature matrix; V is the value feature matrix.

4. The method according to claim 3, characterized in that: Also includes: In the multi-cluster attention mechanism calculation unit corresponding to the multi-cluster attention module, obtaining the Q linear mapping feature extraction result, the K linear mapping feature extraction result and the V linear mapping feature extraction result; Calculation data flow method and formula through improved attention mechanism , and get the calculation results of QKV multi-cluster attention mechanism ; Among them, scale is the external scaling factor; mask is the mask; Softmax is the transcendental function; mask corresponds to the mask matrix.

5. The method according to claim 4, characterized in that The large language model is based on a modular reasoning calculation using an AI-specific acceleration chip; the storage architecture of the AI-specific acceleration chip includes on-chip private storage, on-chip shared storage, and external storage; wherein the size of the external storage is larger than the on-chip shared storage, and the size of the on-chip shared storage is larger than the on-chip private storage; In the feature extraction calculation unit corresponding to the multi-cluster attention module, after performing linear mapping QKV feature extraction on the input text token sequence feature vector to obtain Q linear mapping feature extraction results, K linear mapping feature extraction results and V linear mapping feature extraction results respectively, it also includes: In searching for a feature vector sequence in an input text token sequence, a feature vector of an input text token sequence is obtained in sequence; Performing matrix sub-block segmentation on the first input text token sequence feature vector to obtain matrix sub-block segmentation results of each first input text token sequence feature vector; The linear layer weight tensors in the Q linear mapping feature extraction results, K linear mapping feature extraction results, and V linear mapping feature extraction results are respectively , and Perform matrix sub-block segmentation, and perform data flow arrangement processing through an improved linear layer calculation data flow method according to each first input text token sequence feature vector matrix sub-block segmentation result, and transfer data between on-chip private storage, on-chip shared storage and external storage, traverse all first input text token sequence feature vector matrix sub-block segmentation results and all linear layer weight tensor matrix sub-block segmentation results, and obtain the first input text token sequence feature vector data flow arrangement result; Return to executing the operation of searching for a feature vector sequence in the input text token sequence, and sequentially obtaining an input text token sequence feature vector in order, until all input text token sequence feature vectors are traversed.

6. The method according to claim 5, characterized in that The method and formula for calculating data flow through the improved attention mechanism , and obtain the calculation results of the QKV multi-cluster attention mechanism, including: Perform matrix sub-block segmentation on the query feature matrix, the key feature matrix, the value feature matrix, and the mask matrix respectively, and obtain the sub-block segmentation results of each query feature matrix, the sub-block segmentation results of each key feature matrix, the sub-block segmentation results of each value feature matrix, and the sub-block segmentation results of each mask matrix; Calculation data flow method and formula through improved attention mechanism , the query feature matrix sub-block segmentation results, the key feature matrix sub-block segmentation results, the value feature matrix sub-block segmentation results and the mask matrix sub-block segmentation results are arranged and calculated in data streams, and data is transferred between on-chip private storage, on-chip shared storage and external storage until each query feature matrix sub-block segmentation result traverses each key feature matrix sub-block segmentation result and each value feature matrix sub-block segmentation result to obtain the calculation result of the QKV multi-cluster attention mechanism.

7. A large language model modular reasoning computing device, characterized in that: It is executed by a large language model modular reasoning computing system, including: The input text token sequence feature vector determination module is used to obtain the input text token sequence to be calculated in real time and obtain the target token lookup table dictionary corresponding to the text embedding module; wherein the input text token sequence to be calculated is , n represents the total length of the input text token sequence to be calculated; i represents the input text token; the value range of t is [0, n-1]; the target token lookup table dictionary is , N represents the total length of the target token lookup table dictionary, and h represents the feature vector dimension; in the target token lookup table dictionary, feature vector search processing is performed according to each input text token in the input text token sequence to obtain the input text token sequence search feature vector sequence; wherein, the input text token sequence search feature vector sequence is , Represents an input text token sequence search feature vector; if the first configurable normalization calculation unit is the first non-normalization subunit, each input text token sequence search feature vector is determined as an input text token sequence feature vector; if the first configurable normalization calculation unit is the first LayerNorm normalization calculation subunit, the input text token sequence search feature vector sequence is normalized by the LayerNorm normalization calculation formula to obtain the input text token sequence feature vector y; wherein the LayerNorm normalization calculation formula is ; is the external scaling factor, is the external offset factor, is a constant; if the first configurable normalization calculation unit is the first RMSNorm normalization calculation subunit, the input text token sequence is searched for the feature vector sequence and normalized by the RMSNorm normalization calculation formula to obtain the input text token sequence feature vector y; wherein the RMSNorm normalization calculation formula is ;in, ; The linear calculation result determination module of the input text token sequence is used to input the feature vector of the input text token sequence into the multi-layer loop corresponding to the large language model, and loop between the multi-cluster attention module and the multi-layer perceptron module layer by layer until each layer of loop is traversed to obtain the linear calculation result of the input text token sequence; Each input text generated word token determination module is used to input the linear calculation result of the input text token sequence into the linear mapping module for token probability mapping processing to obtain each input text generated word token; Each input text generates a word token feedback module, which is used to feed back the word token generated by each input text to the user.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements a large language model modular reasoning calculation method as described in any one of claims 2-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a large language model modular reasoning calculation method as described in any one of claims 2-6 when executed.

Images