Data processing method and device, computer equipment and program product

Abstract

The invention provides a data processing method and device, computer equipment and a program product, and the method comprises the steps: carrying out the data processing of input data through a processing unit in a calculation unit which is matched with any network layer of a target neural network, and obtaining output data; sending the output data to at least one other computing unit by using the communication unit in the computing unit and the communication units in the other computing units according to communication connection lines and communication configuration files set for the communication units in the different computing units; wherein the other computing units comprise other computing units matched with the network layer or computing units matched with the next network layer; and returning to the step of performing data processing on the input data by using the processing unit in the calculation unit to obtain the output data until the output data of the last calculation unit matched with the last network layer of the target neural network is obtained, and taking the output data as a data processing result.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a data processing method, apparatus, computer equipment, and program product. Background Art

[0002] The operation of large-scale neural network models requires large-scale computing power, which in turn requires large-scale storage. Large-scale storage requires huge data transmission, which will bring huge power consumption and costs. Therefore, the operation of large-scale neural networks may bring huge power consumption and costs. To solve the power consumption and cost problems, it is necessary to solve the problem of large-scale data transmission. Although some solutions have been proposed, these solutions usually reduce communication flexibility, affect communication efficiency, and have obvious disadvantages. Summary of the invention

[0003] The embodiments of the present disclosure at least provide a data processing method, apparatus, computer device, and program product.

[0004] In a first aspect, an embodiment of the present disclosure provides a data processing method, including:

[0005] For a computing unit that matches any network layer of the target neural network, using a processing unit in the computing unit, performing data processing on input data to obtain output data;

[0006] According to the communication lines and communication configuration files set for the communication units in different computing units, the output data is sent to at least one other computing unit by using the communication unit in the computing unit and the communication units in other computing units; wherein the other computing units include other computing units matching the network layer, or computing units matching the next network layer;

[0007] Return to the step of using the processing unit in the computing unit to process the input data to obtain output data, until the output data of the last computing unit matching the last network layer of the target neural network is obtained, and the output data is used as the data processing result.

[0008] In a possible implementation, the network layer includes multiple functional layers;

[0009] The multiple functional layers at least include a normalization processing functional layer located in the attention layer, a matrix-vector multiplication functional layer related to the query matrix, the key matrix and the value matrix, a multi-head functional layer, a matrix-vector multiplication functional layer related to the result output, an accumulation processing functional layer, and a normalization processing functional layer located in the feedforward neural network layer, a matrix-vector multiplication functional layer related to multiple weight matrices, and an accumulation processing functional layer;

[0010] A computing unit matching any network layer of the target neural network is associated with a functional layer among the multiple functional layers.

[0011] In a possible implementation manner, the using a processing unit in the computing unit to process the input data to obtain output data includes:

[0012] In the case where the computing unit is associated with any matrix-vector multiplication functional layer, using a processing unit in the computing unit, performing matrix-vector multiplication processing on the input data and target data stored in a storage unit in the computing unit to obtain output data;

[0013] Among them, the target data includes any preset matrix corresponding to the functional layer matching the computing unit or a sub-matrix obtained by matrix splitting any preset matrix, and the preset matrix includes a query matrix, a key matrix, a value matrix, a result output matrix and a weight matrix.

[0014] In a possible implementation manner, the using the processing unit in the computing unit to process the input data to obtain output data further includes:

[0015] In the case where the computing unit is independent of the matrix-vector multiplication functional layer, the processing unit in the computing unit is used to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data.

[0016] In a possible implementation manner, before using the processing unit in the computing unit to process the input data to obtain the output data, the method further includes:

[0017] In the case where the input data includes output data sent from multiple other computing units, the output data of each other computing unit is received and cached using multiple data receiving interfaces in the communication unit included in the computing unit;

[0018] In response to each of the data receiving interfaces having completed caching the corresponding output data, the calculation unit is used to trigger a reception completion signal to determine that the acquisition of the input data is completed.

[0019] In a possible implementation, the method further includes:

[0020] Performing functional testing and performance testing on the computing unit;

[0021] In case the detection fails, the computing unit is replaced by a spare computing unit.

[0022] In a possible implementation, the weight matrix includes a feature mapping matrix, a feature transformation matrix, and a feature compression matrix;

[0023] In the case where the preset matrix is ​​any one of the query matrix, the key matrix, the value matrix, the feature mapping matrix, and the feature transformation matrix, the submatrix included in the target data is a matrix obtained by performing row splitting on the preset matrix;

[0024] In the case that the preset matrix is ​​any one of a result output matrix and a feature compression matrix, the submatrix included in the target data is a matrix obtained by performing column splitting on the preset matrix.

[0025] In a possible implementation manner, if the preset matrix is ​​split into multiple sub-matrices, the multiple sub-matrices are respectively stored in storage units of different computing units;

[0026] The sizes of each sub-matrix corresponding to a preset matrix are related to at least one of the size of the preset matrix, the storage space of the storage unit in the calculation unit, and the computing capability of the processing unit in the calculation unit.

[0027] In a possible implementation manner, the using the processing unit in the computing unit to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain the output data includes:

[0028] In the case where the computing unit is a computing unit matching a multi-head functional layer in the attention layer, using a communication unit in the computing unit to obtain a matrix cache associated with a key matrix and a value matrix from a double rate DDR storage unit corresponding to the computing unit;

[0029] The processing unit is used to perform rotation position encoding processing on the input data and the matrix cache according to the network function of the multi-head functional layer to obtain output data.

[0030] In a possible implementation, when the computing unit is a computing unit matching an accumulation processing functional layer in an attention layer, the input data includes first output data sent by the last computing unit matching the previous network layer, and second output data sent by a computing unit matching a matrix-vector multiplication functional layer in the attention layer related to the result output;

[0031] The using the processing unit in the computing unit to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data includes:

[0032] Utilizing the processing unit in the computing unit, the first output data and the second output data are accumulated and processed according to the network function of the accumulation processing function layer in the attention layer to obtain output data.

[0033] In a possible implementation, when the computing unit is a computing unit matching an accumulation processing functional layer in a feedforward neural network layer, the input data includes third output data sent by a computing unit matching a matrix-vector multiplication functional layer associated with a feature compression matrix, and fourth output data sent by a computing unit matching an accumulation processing functional layer in an attention layer;

[0034] The using the processing unit in the computing unit to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data includes:

[0035] The processing unit in the computing unit is used to perform accumulation processing on the third output data and the fourth output data according to the network function of the accumulation processing function layer in the feedforward neural network layer to obtain output data.

[0036] In a possible implementation manner, when the computing unit is a computing unit matched with a matrix-vector multiplication function layer associated with a feature compression matrix, the input data includes fifth output data sent by a computing unit matched with a matrix-vector multiplication function layer associated with a feature mapping matrix, and sixth output data sent by a computing unit matched with a matrix-vector multiplication function layer associated with a feature change matrix;

[0037] The method of using the processing unit in the computing unit to perform matrix-vector multiplication processing on the input data and the target data stored in the storage unit in the computing unit to obtain output data includes:

[0038] The processing unit in the computing unit is used to perform matrix-vector multiplication processing on the fifth output data, the sixth output data and the target data stored in the storage unit in the computing unit to obtain output data.

[0039] In a possible implementation, the sending the output data to at least one other computing unit by using the communication unit in the computing unit and the communication units in other computing units according to the communication lines and communication configuration files set for the communication units in different computing units includes:

[0040] Determine a target functional layer having a communication relationship with the functional layer matched with the computing unit;

[0041] Determining each other computing unit associated with the target functional layer;

[0042] According to the communication connections and communication configuration files set for the communication units in different computing units, the output data is sent to each other computing unit respectively by utilizing the data sending interface included in the communication unit in the computing unit and the data receiving interface included in the communication units in each other computing unit.

[0043] In a second aspect, an embodiment of the present disclosure further provides a data processing device, including:

[0044] A processing module, for processing input data using a processing unit in a computing unit that matches any network layer of a target neural network, to obtain output data;

[0045] A sending module, configured to send the output data to at least one other computing unit by using the communication unit in the computing unit and the communication unit in other computing units according to the communication connection and communication configuration file set for the communication units in different computing units; wherein the other computing units include other computing units matching the network layer, or computing units matching the next network layer;

[0046] The loop module is used to return to the step of using the processing unit in the computing unit to process the input data to obtain output data, until the output data of the last computing unit matching the last network layer of the target neural network is obtained, and the output data is used as the data processing result.

[0047] In a third aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0048] In a fourth aspect, an optional implementation of the present disclosure further provides a computer program product, including a computer program, which, when executed, implements the above-mentioned first aspect, or the steps in any possible implementation of the first aspect.

[0049] For a description of the effects of the above-mentioned data processing apparatus, computer equipment, and computer program product, please refer to the description of the above-mentioned data processing method, which will not be repeated here.

[0050] The data processing method, device, computer equipment and program product provided by the embodiments of the present disclosure can deploy a communication unit in a computing unit, which can achieve data reception for the computing unit and flexible communication interconnection between multiple computing units at a relatively small unit area cost, thereby improving the interconnection performance and communication flexibility between multiple computing units. Moreover, compared with the prior art of using a graphics processor to complete large-scale calculations in the model, by pre-setting at least one matching computing unit for each network layer of the target neural network and ensuring that the computing unit can focus on the reasoning calculation of the corresponding network layer, it is achieved that the reasoning calculation of the model is completed by replacing the expensive graphics processor with each low-cost computing unit, which not only reduces the reasoning cost of the large model, but also reduces the reasoning power consumption of the large model. In addition, by matching different computing units to different target neural networks and setting communication lines and communication configuration files for each matching computing unit, each computing unit can perform data processing in an orderly manner according to the indicated processing order, thereby improving the accuracy of data processing.

[0051] Furthermore, the data processing method, apparatus, computer equipment and program product provided by the embodiments of the present disclosure, since the matrix-vector multiplication operation in the large-scale neural network reasoning process requires the use of frequent operators, and such operations will have a large-scale calculation ratio when calculated by the number of operands, it is sometimes difficult to cache the required huge matrix using a single computing unit. Therefore, the present application proposes to split the matrix using row or column splitting methods, and pre-store the sub-matrices obtained by the splitting in different computing units, while ensuring that a computing unit only resides in the sub-matrix corresponding to a certain type of matrix. This effectively solves the calculation problem of large-scale matrices and reduces the power consumption caused by large-scale data transmission, and enables each computing unit to focus on the calculation of the corresponding network layer, thereby ensuring the accuracy of model reasoning.

[0052] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0054] Figure 1A flow chart of a data processing method provided by an embodiment of the present disclosure is shown;

[0055] Figure 2 A schematic diagram of the structure of a computing unit provided by an embodiment of the present disclosure is shown;

[0056] Figure 3 A schematic diagram of the structure of a network layer provided by an embodiment of the present disclosure is shown;

[0057] Figure 4 A schematic diagram of a data processing device provided by an embodiment of the present disclosure is shown;

[0058] Figure 5 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0060] In addition, the terms "first", "second", etc. in the description and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0061] The "multiple or several" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0062] Research has found that in order to solve the power consumption and cost problems caused by large-scale neural network models during data inference, the traditional method has proposed using multiple chips to retain data in advance to solve the data transmission problem. Although this method can reduce power consumption and cost, it reduces the flexibility of communication and has certain disadvantages.

[0063] Based on the above research, the present disclosure provides a data processing method, device, computer equipment and program product. By deploying a communication unit in a computing unit, data reception for the computing unit can be achieved at a relatively small unit area cost, and flexible communication and interconnection between multiple computing units can be achieved, thereby improving the interconnection performance and communication flexibility between multiple computing units. Moreover, compared with the prior art of using a graphics processor to complete large-scale calculations in the model, by pre-setting at least one matching computing unit for each network layer of the target neural network and ensuring that the computing unit can focus on the reasoning calculation of the corresponding network layer, the reasoning calculation of the model can be completed by replacing the expensive graphics processor with each low-cost computing unit, which not only reduces the reasoning cost of the large model, but also reduces the reasoning power consumption of the large model. In addition, by matching different computing units for different target neural networks and setting communication lines and communication configuration files for each matching computing unit, each computing unit can perform data processing in an orderly manner according to the indicated processing order, thereby improving the accuracy of data processing.

[0064] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0065] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0066] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0067] To facilitate understanding of this embodiment, a data processing method disclosed in an embodiment of the present disclosure is first introduced in detail. The executor of the data processing method provided in the embodiment of the present disclosure is generally a terminal device or other processing device with certain computing capabilities, wherein the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a handheld device, a computer device, a computing unit (such as a chip), etc.; in some possible implementations, the data processing method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0068] The data processing method provided by the embodiment of the present disclosure is described below by taking the execution subject as a computing unit as an example.

[0069] like Figure 1 As shown, it is a flowchart of a data processing method provided by an embodiment of the present disclosure, which may include the following steps:

[0070] S101: For a computing unit that matches any network layer of a target neural network, use a processing unit in the computing unit to process input data to obtain output data.

[0071] Here, the target neural network can be a network model with any reasoning function, for example, the target neural network can be a large language model, a large language model based on Transformer, other large-scale network models based on Transformer, etc. That is, the data processing method provided in the embodiment of the present disclosure can be applied to the large language model reasoning scenario.

[0072] The target neural network can be a single-layer network model including one network layer, or a multi-layer network model including multiple network layers, which is not specifically limited in the embodiments of the present disclosure. For each network layer, at least one computing unit that matches the network layer can be predetermined, and the processing order of each computing unit is related to the reasoning order of the target neural network. For example, a matching computing unit cluster (system) can be set up for the target neural network in advance, and the cluster includes computing units that match each network layer of the target neural network, and each computing unit is relatively independent and completely consistent in design. The control and coordination between the computing units is achieved through different pre-set communication lines and communication configuration files, and ultimately realizes the functions of a large-scale neural network model.

[0073] The computing unit in the embodiment of the present disclosure is a small-scale unit, and its cost, power consumption, storage capacity, etc. are lower than those of a conventional image processor. However, by setting a matching computing unit cluster for the target neural network, the function of the image processor is replaced during model reasoning, thereby reducing power consumption and cost while ensuring model reasoning. Optionally, the computing unit in the embodiment of the present disclosure can be a pre-developed chip.

[0074] like Figure 2As shown, it is a schematic diagram of the structure of a computing unit provided in an embodiment of the present disclosure, and the computing unit includes a processing unit, a storage unit and a communication unit. The processing unit is a unit with a computing function, which can be used for data calculation. For example, the processing unit can realize matrix-vector multiplication calculation, nonlinear calculation, element-wise calculation, etc. Optionally, the processing unit can be implemented by in-memory calculation or near-memory calculation. The storage unit is used for data storage, for example, the sub-matrix described later can be pre-stored, and the storage unit can be, for example, a static random access memory (SRAM).

[0075] The communication unit is used to realize the communication between different computing units, that is, the computing units can be organized together by communication units to jointly solve large-scale computing problems; in terms of overall structure and area, the storage unit occupies a larger area, followed by the processing unit and the computing unit. The core of this disclosure is how to organize the computing units through the design of communication to jointly complete the single-layer and multi-layer reasoning of the large language model. Figure 2 Specifically, it may include an inter-chip interconnection interface and a double data rate (DDR) interface. The inter-chip interconnection interface is used to realize the communication interconnection between computing units, for example, it may include a serial peripheral interface (SPI) and multiple communication interfaces. Multiple communication interfaces may be, for example, N groups of sending interfaces and N groups of receiving interfaces, or N groups of transceiver interfaces, where the number N may be 8, 16, 32, etc. The SPI interface is used to realize the interconnection between computing units. For example, different computing units can be connected according to different interface functions through the SPI interface, so as to realize communication between units. The sending interface is used to realize data transmission, and the receiving interface is used to realize the data interface. The transceiver interface is an interface that can flexibly configure the transceiver mode. For any group of transceiver interfaces, the transceiver mode of the interface can be controlled by software. For example, when the computing unit needs to receive data, the transceiver interface can be configured as a receiving mode through software; when the computing unit needs to send data, the transceiver interface can be configured as a sending mode through software. The DDR interface is used to connect to a storage unit disposed outside the computing unit, so as to store some data processed by the computing unit in the external storage unit, wherein the external storage unit may be, for example, a DDR storage unit.

[0076] The communication unit can send the calculation results of the processing unit to one or more transmission interfaces, and send the calculation results to other computing units through the communication connection (such as SPI connection) on the circuit board corresponding to the computing unit and the receiving interface on other computing units in advance. In addition, different master / slave relationships can be configured for the same computing unit through the connection on the circuit board, thereby realizing the orderly flow of data between multiple computing units.

[0077] The target neural network may include one or more network layers, and each network layer has at least one matching computing unit. For any computing unit matched by any network layer, the input data is the data input to the computing unit, and the data can be processed by the processing unit in the computing unit. The input data of computing units matched by different network layers are different. For example, the input data of a computing unit may be at least one of the output data sent by other computing units matched by the previous network layer through the internal communication unit, the initial input data for the target neural network, and the output data sent by other computing units matched by the network layer through the communication unit. Among them, the initial input data can be regarded as the original data that needs to be input to the target neural network.

[0078] In one embodiment, each network layer of the target neural network may include multiple functional layers, and different functional layers have different network functions. Specifically, the multiple functional layers at least include a normalization processing functional layer (norm) located in the attention layer, a matrix-vector multiplication functional layer (matmul(QKV)) related to the query matrix, the key matrix, and the value matrix, a multi-head functional layer (multihead), a matrix-vector multiplication functional layer (matmul(Output)) related to the result output, an accumulation processing functional layer (Add), and a normalization processing functional layer (Norm) located in the feedforward neural network layer (Feedforward Neural Network, FNN), a matrix-vector multiplication functional layer related to multiple weight matrices, and an accumulation processing functional layer (Add). The weight matrix may specifically include a feature mapping matrix (hereinafter referred to as the W1 matrix), a feature transformation matrix (hereinafter referred to as the W3 matrix) and a feature compression matrix (hereinafter referred to as the W2 matrix), and the matrix-vector multiplication function layers respectively associated with the multiple weight matrices may include a matrix-vector multiplication function layer (matmul(W1)) associated with the W1 matrix, a matrix-vector multiplication function layer (matmul(W2)) associated with the W2 matrix, and a matrix-vector multiplication function layer (matmul(W3)) associated with the W3 matrix. A computing unit that matches any network layer of the target neural network is associated with a functional layer among the multiple functional layers. That is, in the embodiment of the present disclosure, each functional layer may have at least one matching computing unit, and each computing unit that matches any network layer of the target neural network is a computing unit respectively associated with each functional layer in the network layer. The computing units that match each functional layer can be pre-set before using the target neural network for model inference.

[0079] like Figure 3 As shown, it is a schematic diagram of the structure of a network layer provided by an embodiment of the present disclosure, in Figure 3In , a schematic diagram of the structure of the current network layer is shown. Last Block can be understood as the previous network layer of the current network layer, and next Block can be understood as the next network layer of the current network layer. The current network layer includes the Attention layer and the FFN layer. The Attention layer includes the input (Input) layer, the Norm layer, the matmul (QKV) layer, the multihead layer, the matmul (Output) layer, and the Add layer in sequence. The processing order and data input relationship between each layer are shown by the arrows. Among them, in the case where the computing unit matches the input layer of the non-first network layer of the target neural network, the input data can be the data sent by the computing unit that matches the previous network layer of the non-first network layer through the internal communication unit, and the data can be the output data of the computing unit that matches the previous network layer. For example, in Figure 3 In , the input data of the Input layer can be the output data sent by the computing unit that matches the previous network layer through the internal communication unit. In the case where the computing unit matches the input layer of the first network layer of the target neural network, the input data can be the initial input data for the target neural network. In the case where the computing unit matches the non-input layer of any network layer of the target neural network, the input data can be the output data sent by the computing unit that matches other functional layers of the network layer through the internal communication unit. For example, in Figure 3 In the example, the input data of any computing unit matched by the matmul(QKV) layer can be the output data sent by the computing unit matched by the Norm layer. Figure 3 The input data of the computing unit matched by the Add layer in the attention layer may include the output data of the Input layer and the output data sent by the computing unit matched by the matmul (Output) layer through the communication unit.

[0080] The FFN layer includes the Norm layer, matmul(W1) layer, matmul(W3) layer, matmul(W2) layer, Add layer and output layer in sequence. The processing order and data input relationship between the layers are shown by arrows. The input of the FFN layer is the output of the entire Attention layer, and the output of the FFN layer is the output of the entire network layer. The input data of the matmul(W3) layer may include the output data sent by the matching computing unit of the Norm layer, and the input data of the matmul(W2) layer may include the output data sent by the matching computing unit of the matmul(W1) layer and the output data sent by the matching computing unit of the matmul(W3) layer. The input data of the Add layer may include the output data of the Attention layer and the output data sent by the matching computing unit of the Add layer of the FFN layer. It can be understood that the Input layer and the Output layer are only used for data input and output, and no data calculation is performed, so the Input layer and the Output layer may not be set with matching computing units.

[0081] Exemplarily, a corresponding computing unit cluster can be set up in advance for the target neural network, and the communication unit can be used to realize the interconnection between the various computing units. Then, before executing S101, for any computing unit that matches any network layer of the target neural network, the communication unit in the computing unit can be used to obtain input data that matches the computing unit; the input data includes at least one of the output data sent by other computing units that match the previous network layer through the communication unit, the initial input data for the target neural network, and the output data sent by other computing units that match the network layer through the communication unit. For example, for any computing unit, the data receiving interface in the communication unit in the computing unit can be used to obtain input data related to the computing unit. Furthermore, the processing unit in the computing unit can be used to process the input data to obtain output data.

[0082] The data processing that the processing unit in the computing unit needs to perform is related to the network function of the functional layer matched by the computing unit. The data processing may include, for example, matrix-vector multiplication, accumulation processing, rotation position encoding, nonlinear calculation, etc.

[0083] In a specific implementation, for any computing unit, after obtaining input data using the communication unit of the computing unit, the processing unit in the computing unit can continue to be used to perform corresponding data processing on the input data according to the network function of the functional layer matched by the computing unit, to obtain a data processing result, and use the result as the output data of the computing unit.

[0084] S102: According to the communication connections and communication configuration files set for the communication units in different computing units, the output data is sent to at least one other computing unit using the communication unit in the computing unit and the communication units in other computing units; wherein the other computing units include other computing units matching the network layer, or computing units matching the next network layer.

[0085] Here, the communication profile is used to indicate the communication parameters and communication configuration between the computing units. The communication profile may include a communication profile corresponding to each computing unit, or it may be an overall profile for all computing units that match the target neural network. Different target neural networks require different computing unit clusters to be set, and the communication connections and communication profiles between the computing units in the computing unit clusters are different. That is, for different target neural networks, different numbers of computing units may be set in advance, and different communication connections and communication profiles may be set for each computing unit. Exemplarily, for any target neural network, the number of computing units, communication connections and communication profiles between the computing units that need to be set for the target neural network may be determined according to the function of the target neural network.

[0086] Other computing units are computing units that need to perform the next step of computing processing. Other computing units can be determined based on pre-set communication connections and communication configuration files. For example, after the computing units matched in the Norm layer obtain the output data, other computing units can be computing units matched with the matmul (QKV) layer. For another example, after the Output layer of the FFN layer obtains the output data, other computing units can be computing units matched with the input layer of the next network layer. In this way, data processing can be realized through direct communication between computing units, which can improve the efficiency of data processing.

[0087] In specific implementation, after any computing unit determines the output data, the other computing units that need to perform the next step of computing processing can be determined based on the communication configuration file, the preset communication connection matched by the computing unit, the functional layer matched by the computing unit, and the processing order between the functional layers in the target neural network. Then, the communication unit in the computing unit and the communication units in each other computing unit can be used to send the output data of the computing unit to each other computing unit respectively, so that each other computing unit can perform the next step of model reasoning.

[0088] S103: Return to the step of using the processing unit in the computing unit to process the input data to obtain output data, until the output data of the last computing unit matching the last network layer of the target neural network is obtained, and the output data is used as the data processing result.

[0089] Here, the last computing unit matched by the last network layer can be specifically the Output layer of the last network layer in the target neural network. The data processing result is the final processing result output by the target neural network after performing data inference on the initial input data.

[0090] Exemplarily, after the output data of any computing unit is sent to other computing units, the output data can be regarded as the input data of other computing units, and then, for each other computing unit, the step of S101 can be returned to be executed, so as to continue to use other computing units to complete further reasoning processing of the data. After obtaining the output data, other computing units can continue to send the output data to new other computing units through their internal communication units for further processing, and the data processing results output by the Output layer of the last network layer are obtained after being processed in sequence by various computing units matching the target neural network.

[0091] It is understandable that before the computing unit receives the data or before the computing unit sends the data to other computing units, the input data can also be processed separately using the set additional processing unit. The separate processing here is only for the input data itself, and no interaction between the input data and other data will occur. For example, the separate processing can include square root calculation, natural exponential calculation, natural logarithm calculation, silicon rectified linear unit (silu) calculation, etc. of the input data.

[0092] In this way, by deploying the communication unit in the computing unit, it is possible to achieve data reception for the computing unit and flexible communication interconnection between multiple computing units at a relatively small unit area cost, thereby improving the interconnection performance and communication flexibility between multiple computing units. Moreover, compared with the prior art of using a graphics processor to complete large-scale calculations in the model, by pre-setting at least one matching computing unit for each network layer of the target neural network and ensuring that the computing unit can focus on the reasoning calculation of the corresponding network layer, it is possible to use each low-cost computing unit to replace the expensive graphics processor to complete the reasoning calculation of the model, thereby reducing the reasoning cost of the large model and the reasoning power consumption of the large model. In addition, by matching different computing units to different target neural networks and setting communication lines and communication configuration files for each matching computing unit, each computing unit can perform data processing in an orderly manner according to the indicated processing order, thereby improving the accuracy of data processing.

[0093] In one embodiment, for the specific implementation process of S101, different implementation methods may be adopted depending on whether the computing unit is related to the matrix-vector multiplication function layer. The matrix-vector multiplication function layer may include, for example, the matmul(QKV) layer, matmul(Output) layer, matmul(W1) layer, matmul(W3) layer, and matmul(W2) layer mentioned above.

[0094] Specifically, when a computing unit is associated with any matrix-vector multiplication functional layer, a processing unit in the computing unit is used to perform matrix-vector multiplication processing on input data and target data stored in a storage unit in the computing unit to obtain output data; wherein the target data includes any preset matrix corresponding to a functional layer matching the computing unit or a sub-matrix obtained by matrix splitting any preset matrix, and the preset matrices include a query matrix, a key matrix, a value matrix, a result output matrix and a weight matrix.

[0095] Here, for any network layer, any functional layer in the network layer has a corresponding network function. Moreover, for any matrix-vector multiplication functional layer, there is a corresponding preset matrix, and the matrix size of the preset matrix is ​​related to the processing power of the target neural network. Matrix-vector multiplication refers to the process of multiplying a vector of length N and a matrix of size M×N to obtain a vector of length M. In this process, the size of the matrix is ​​MxN. When M and N are very large (such as 10000X10000), it is often difficult to cache such a large matrix on the chip. Therefore, it is necessary to split it into multiple matrices, calculate the partial sum of multiple matrices separately, and realize multiplication and addition calculations.

[0096] Exemplarily, for the matmul(QKV) layer, the corresponding preset matrix may include a preset query matrix (i.e., Q matrix), a preset key matrix (i.e., K matrix), and a preset value matrix (i.e., V matrix). For the matmul(Output) layer, the corresponding preset matrix may include a preset result output matrix (i.e., Output matrix). For the matmul(W1) layer, the corresponding preset matrix may include a preset W1 matrix, for the matmul(W2) layer, the corresponding preset matrix may include a preset W2 matrix, and for the matmul(W3) layer, the corresponding preset matrix may include a preset W3 matrix.

[0097] Taking the Q matrix, K matrix and V matrix as examples, the Q matrix, K matrix and V matrix can each include at least one matching computing unit. For example, for the Q matrix, if the computing unit can process the matrix-vector multiplication operation corresponding to the matrix, the matrix can be deployed in a computing unit, and the computing unit is only used to process operations related to the Q matrix. If the computing unit cannot process the matrix-vector multiplication operation corresponding to the matrix, the Q matrix can be split into multiple sub-matrices according to the processing capability of the computing unit, and the multiple sub-matrices are deployed in different computing units. These computing units are only used to process operations related to the deployed sub-matrices, and the processing results of these computing units are accumulated to obtain the complete results corresponding to the Q matrix.

[0098] Therefore, for any computing unit related to any matrix-vector multiplication functional layer, the storage unit of the computing unit may pre-store a preset matrix corresponding to the matrix-vector multiplication functional layer or a sub-matrix split from the preset matrix, wherein the preset matrix or sub-matrix stored in the storage unit may be referred to as target data.

[0099] Exemplarily, for any calculation unit that matches the Q matrix, the processing unit in the calculation unit can be used to perform matrix-vector multiplication on the input data and the target data (Q matrix or sub-matrix of Q matrix) stored in the storage unit of the calculation unit to obtain output data. For any calculation unit that matches the K matrix, the processing unit in the calculation unit can be used to perform matrix-vector multiplication on the input data and the target data (K matrix or sub-matrix of K matrix) stored in the storage unit of the calculation unit to obtain output data. For any calculation unit that matches the V matrix, the processing unit in the calculation unit can be used to perform matrix-vector multiplication on the input data and the target data (V matrix or sub-matrix of V matrix) stored in the storage unit of the calculation unit to obtain output data. For any calculation unit that matches the Output matrix, the processing unit in the calculation unit can be used to perform matrix-vector multiplication on the input data and the target data (Output matrix or sub-matrix of Output matrix) stored in the storage unit of the calculation unit to obtain output data. For any computing unit that matches the Wx matrix, the processing unit in the computing unit can be used to perform matrix-vector multiplication on the input data and the target data (Wx matrix or a submatrix of the Wx matrix) stored in the storage unit of the computing unit to obtain output data. Wherein, the value of x is 1, 2 or 3.

[0100] In this way, each computing unit is related to a preset matrix corresponding to a functional layer, and is only used to perform operations related to the stored target data in a certain reasoning operation, which refines the processing operations of each computing unit and reduces the processing difficulty of the computing unit.

[0101] In another embodiment, when the computing unit is independent of the matrix-vector multiplication functional layer, the processing unit in the computing unit is used to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data.

[0102] Here, the computing units independent of the matrix-vector multiplication function layer may include computing units matched by the Norm layer, computing units matched by the multihead layer, computing units matched by the Add layer, and the like.

[0103] The network functions may include, for example, the normalization processing function of the Norm layer, the multi-head attention processing function of the multihead layer, and the accumulation processing function of the Add layer.

[0104] For example, for any computing unit matching the Norm layer, the processing unit in the computing unit can be used to normalize the input data to obtain output data. For any computing unit matching the multihead layer, the processing unit in the computing unit can be used to perform multi-head attention processing on the input data to obtain output data. For any computing unit matching the Add layer, the processing unit in the computing unit can be used to accumulate the input data to obtain output data.

[0105] In one embodiment, the output data of a computing unit may need to be sent to multiple computing units matched by the next functional layer. For example, the output data of the computing unit matched by the Norm layer may need to be sent synchronously to the computing units in the matmul (QKV) layer that match the Q matrix, K matrix, and V matrix respectively. In this case, the communication unit in the embodiment of the present disclosure may have a broadcast function, based on which a copy of the output data can be sent to the communication units in multiple computing units, thereby ensuring that each computing unit can use the output data. Similarly, a computing unit may need to synchronously receive output data from multiple other computing units. For example, for the computing unit matched by the multihead layer, it may need to synchronously receive output data from computing units that match the Q matrix, K matrix, and V matrix respectively. Therefore, the step of obtaining input data for the computing unit in S101 above can be implemented as follows:

[0106] In the case where the input data includes output data sent from multiple other computing units, multiple data receiving interfaces in the communication unit included in the computing unit are used to receive and cache the output data of each computing unit respectively; in response to each data receiving interface having completed caching the corresponding packet data, the computing unit is used to trigger a reception completion signal to determine that the input data acquisition is completed.

[0107] Here, for any computing unit, each data receiving interface included in the communication unit in the computing unit has a communication caching function. In the case where multiple groups of data receiving interfaces of a computing unit need to simultaneously receive output data from multiple computing units matched with a certain functional layer, the output data of multiple computing units can be received separately using multiple data receiving interfaces in the communication unit according to the communication connection and the communication configuration file. That is, the output data of a computing unit is received using one data receiving interface. At the same time, for each data receiving interface, the received output data can be received and cached. After each data receiving interface has cached the output data of the corresponding computing unit, the computing unit can trigger a reception completion signal to use the signal to indicate that the synchronous reception of the output data from multiple computing units is completed, thereby achieving complete acquisition of the input data.

[0108] That is, in the embodiment of the present disclosure, when the multiple data receiving interfaces of the computing unit simultaneously receive data, a receiving completion signal indicating that the synchronous receiving of multiple data sets is completed is also provided. The receiving completion signal is triggered only when the data corresponding to the multiple data receiving interfaces are all received.

[0109] Optionally, for the output data of any computing unit, when the computing unit sends the output data to other computing units, the computing unit may also split the output data into multiple groups of data, so that other computing units will use multiple groups of data receiving interfaces to receive each group of data respectively and receive and cache. When multiple groups of data are received, a reception completion signal is triggered.

[0110] In one embodiment, in order to achieve a bottom-up operation, the embodiment of the present disclosure can also perform a board-level redundancy design, that is, at least one spare computing unit can be set up to achieve timely replacement when a computing unit fails. Therefore, the embodiment of the present disclosure can also perform functional testing and performance testing on the computing unit; if the test fails, the computing unit is replaced with a spare computing unit.

[0111] Here, the functional test is used to detect whether the function of the computing unit is normal, and the performance test is used to detect whether the performance of the computing unit can meet the preset standard.

[0112] Exemplarily, a detection program may be provided, and for any computing unit that matches the target neural network, the detection program may be used to perform functional detection and performance detection on the computing unit. If both detections pass, it indicates that the computing unit can be used normally. If one of the two detections fails, it may indicate that the computing unit is faulty and the detection fails. At this time, the computing unit may be replaced with a spare computing unit through the SPI interface and routing protocol to ensure the normal operation of subsequent model reasoning. Optionally, the spare computing unit may be a computing unit that can only receive a single input and a single output, and has poorer flexibility of use than the replaced computing unit.

[0113] In one embodiment, the weight matrix includes a feature mapping matrix, a feature transformation matrix, and a feature compression matrix. When the preset matrix is ​​any one of a query matrix, a key matrix, a value matrix, a feature mapping matrix, and a feature transformation matrix, the submatrix included in the target data is a matrix obtained by splitting the preset matrix into rows. When the preset matrix is ​​any one of a result output matrix and a feature compression matrix, the submatrix included in the target data is a matrix obtained by splitting the preset matrix into columns.

[0114] That is, the embodiment of the present disclosure provides two matrix splitting methods, namely row splitting (i.e. splitting by row) and column splitting (i.e. splitting by column), and different splitting methods will be used for the corresponding preset matrices in different functional layers. For row splitting, the communication unit can be used to broadcast the output data of other computing units, so as to send the output data to the computing unit storing each submatrix obtained by row splitting. Each computing unit can complete M rows of matrix-vector multiplication operations and output M results. Among them, the number of M is related to the number of rows included in the submatrix. Then, the M results output by each computing unit can be input to the next node (i.e. other computing units) to realize the splicing of the M results output by each computing unit, and obtain the matrix-vector multiplication result for the preset matrix.

[0115] For column splitting, the communication unit can be used to broadcast the output data of other computing units, so as to realize sending the output data to the computing unit storing each sub-matrix obtained by column splitting. Each computing unit can use the stored sub-matrix to complete the matrix-vector multiplication operation, obtain the sub-result, and then send the sub-result to the adjacent computing unit (i.e., the computing unit matched by the Add layer) for accumulation. If it is a one-way accumulation, the matrix-vector multiplication result for the preset matrix can be obtained by accumulating the sub-results sent by the computing units storing the sub-matrices of each column split. For each computing unit storing the sub-matrix of column splitting, after the computing unit sends the sub-result to the adjacent computing unit, the calculation result of the computing unit will be cleared. In this way, after the one-way transmission of each computing unit, the accurate matrix-vector multiplication result can be accumulated.

[0116] It is understandable that the embodiments of the present disclosure also involve splitting rows first and then columns, wherein row splitting can give partial numerical values ​​of the result, and the output of row splitting is used as the input of column splitting. After the column splitting completes the calculation, the results of column splitting are accumulated to obtain the final result of matrix-vector multiplication. Among them, row first and column later can be reflected in that after the calculation units respectively matching the Q matrix, K matrix and V matrix obtain the output data, it will eventually be transmitted to the calculation unit matching the Output matrix for further processing; and it can be reflected in that after the calculation units respectively matching the W1 matrix and the W3 matrix obtain the output data, it will be transmitted to the calculation unit matching the W2 matrix for further processing.

[0117] In one embodiment, if a preset matrix is ​​split into multiple sub-matrices, the multiple sub-matrices are stored in storage units of different computing units respectively; wherein the storage unit of a computing unit will only store one sub-matrix related to a preset matrix, and the computing unit will only be used to perform matrix-vector multiplication operations on the sub-matrix, and will not perform matrix-vector multiplication operations on other sub-matrices. The size of each sub-matrix corresponding to a preset matrix is ​​related to at least one of the size of the preset matrix, the storage space of the storage unit in the computing unit, and the computing power of the processing unit in the computing unit.

[0118] Taking the Q matrix as an example, it is possible to determine how to split the Q matrix into rows according to one or more of the size of the Q matrix, the storage space of the storage unit in the computing unit, and the computing power of the processing unit in the computing unit, so as to obtain multiple sub-matrices corresponding to the Q matrix. The size of each sub-matrix is ​​smaller than the storage space of the storage unit in the computing unit, and the calculation can be completed when the processing unit in the computing unit performs matrix-vector multiplication on the sub-matrix and the input data.

[0119] Taking the W2 matrix as an example, how to split the W2 matrix into columns can be determined based on one or more of the size of the W2 matrix, the storage space of the storage unit in the computing unit, and the computing power of the processing unit in the computing unit, so as to obtain multiple sub-matrices corresponding to the W2 matrix.

[0120] Continue to refer Figure 3 , briefly explain the principle of matrix splitting. Figure 3In the example, the input data is an 8×1 vector. After the 8×8 preset matrix is ​​split into rows, a 2×8 submatrix (such as a submatrix filled with black) is obtained. In this way, the 8×8 preset matrix will be split into 4 submatrices and stored in the storage space of 4 different computing units. After broadcasting the 8×1 vector to these 4 computing units, each computing unit can perform a matrix-vector multiplication operation on the 8×1 vector and the 2×8 submatrix to obtain the output data (i.e., a 2×1 vector). By splicing the output data of the 4 computing units, the 8×1 matrix-vector multiplication result can be obtained.

[0121] For column splitting, the input data is the output data of the previous computing unit (i.e., a 2×1 vector). After the 8×8 preset matrix is ​​split into columns, an 8×2 sub-matrix is ​​obtained. Such a sub-matrix includes 4 and is stored in the storage space of 4 different computing units. After broadcasting the 2×1 vector to these 4 computing units, each computing unit can perform a matrix-vector multiplication operation on the 2×1 vector and the 8×2 sub-matrix to obtain the output data (i.e., an 8×1 vector). By accumulating the output data of the 4 computing units, the final matrix-vector multiplication result can be obtained.

[0122] exist Figure 3 In the process, the Q matrix, K matrix and V matrix in the matmul(QKV) layer are split into rows and stored in storage units in different computing units. The Output matrix in the matmul(Output) layer is split into columns and stored in storage units in different computing units. The W1 matrix in the matmul(W1) layer is split into rows and stored in storage units in different computing units. The W3 matrix in the matmul(W3) layer is split into rows and stored in storage units in different computing units. The W2 matrix in the matmul(W2) layer is split into columns and stored in storage units in different computing units. Afterwards, the data reasoning process can be implemented using each computing unit according to the data reasoning process described in the above embodiments.

[0123] In one embodiment, to facilitate understanding of the embodiments of the present disclosure, the processing of the computing unit that matches some functional layers will be described below. For example, with respect to the step of "using the processing unit in the computing unit to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data", when the computing unit is a computing unit that matches the multi-head functional layer located in the attention layer, the communication unit in the computing unit is used to obtain the matrix cache related to the key matrix and the value matrix from the double-rate DDR storage unit corresponding to the computing unit. Then, the processing unit can be used to perform rotational position encoding processing on the input data and the matrix cache according to the network function of the multi-head functional layer to obtain the output data.

[0124] Here, multihead can be implemented in a single-layer network function layer, and the data processing operations in multihead may include rotation position encoding operations, which mainly perform bitwise multiplication of vectors and vectors, addition and subtraction of vectors, etc. One part of the two-part vector comes from the output of each computing unit matched by the matmul (QKV) layer, and the other part needs to be actively loaded by the computing unit matched by the multihead layer. The data is cache data related to the K matrix and the V matrix, which can be called KVCache. KVCache can be obtained by the computing unit matched by the matmul (QKV) layer during the process of matrix-vector multiplication and sent to the DDR storage unit through the DDR interface for storage. In the embodiment of the present disclosure, there is an association relationship between the computing unit and the DDR storage unit, and multiple computing units can be associated with the same DDR storage unit. The DDR storage unit is located outside the computing unit, and each computing unit can communicate with the associated DDR storage unit through the internal DDR interface.

[0125] Exemplarily, for the computing unit matched with the multihead layer, the computing unit can receive the output data sent by each computing unit matched with the matmul (QKV) layer, and use these output data as input data. At the same time, the DDR interface in the internal communication unit can be used to obtain all the KVCache stored in the unit from the corresponding DDR storage unit. Then, the processing unit can be used to perform rotational position encoding processing on the input data and the obtained KVCache according to the multi-head attention processing function corresponding to the multihead layer to obtain the output data corresponding to the multihead layer. Optionally, when using the computing unit for data reasoning, the embodiment of the present disclosure can also use the Group Query Attention (GQA) method to reduce the KVCache usage in the calculation process.

[0126] Optionally, in the nonlinear spike framework inside the multihead layer, the computing unit matched with the multihead layer can also be used to perform processing such as normalization (softmax) and scalar division on the input data. There may also be matrix-vector multiplication operations inside the multihead layer, but the matrix-vector multiplication operations here may be smaller in scale and can be completed using computing units that perform rotational position encoding. Of course, it can also be implemented using other computing units preset for the multihead layer, or it can also be completed using computing units matched with the matmul (QKV) layer. Here, since the computing units matched with the matmul (QKV) layer have been processed and are idle when the computing units matched with the multihead layer are processing, the calculation can be completed by reusing the idle computing units.

[0127] In one embodiment, when the computing unit is a computing unit that matches the accumulation processing functional layer located in the attention layer, the input data includes the first output data sent by the last computing unit that matches the previous network layer, and the second output data sent by the computing unit that matches the matrix-vector multiplication functional layer related to the result output located in the attention layer,

[0128] Here, combined Figure 3 It can be seen that the previous network layer of the Add layer in the Attention layer is the previous Block. The first output data sent by the last computing unit that matches the previous network layer is the output data of the Output layer in the previous Block, and is also the data of the Input layer of the current network layer in the input. The matrix-vector multiplication function layer related to the result output in the attention layer is the matmul (Output) layer, and the second output data is the output data of the computing unit matched by the matmul (Output) layer. It should be noted that when data transmission between different computing units is involved in the embodiments of the present disclosure, it is necessary to rely on the communication unit set inside the computing unit to complete.

[0129] Exemplarily, for any computing unit that matches the Add layer in the attention layer, the input data of the computing unit includes the output data of each computing unit matched by the matmul(Output) layer and the output data of the Output layer of the previous network layer.

[0130] Regarding the step of "using the processing unit in the computing unit to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data", the processing unit in the computing unit can be used to perform accumulation processing on the first output data and the second output data according to the network function of the accumulation processing functional layer in the attention layer to obtain the output data.

[0131] Exemplarily, for a computing unit matching the Add layer, the computing unit may utilize a processing unit to perform accumulation processing on the first output data and each second output data according to an accumulation function, thereby obtaining output data of the computing unit.

[0132] In another embodiment, when the computing unit is a computing unit matched with an accumulation processing functional layer located in a feedforward neural network layer, the input data includes third output data sent by a computing unit matched with a matrix-vector multiplication functional layer associated with a feature compression matrix, and fourth output data sent by a computing unit matched with an accumulation processing functional layer located in an attention layer.

[0133] Here, combined Figure 3 It can be seen that the third output data can be the output data of each computing unit matching the matmul(W2) layer; the fourth output data is the output data of the computing unit matching the Add layer in the Attention layer.

[0134] Exemplarily, for any computing unit that matches the Add layer in FFN, the input data of the computing unit may include the output data of each computing unit matched by the matmul (W2) layer, and the computing unit matched by the Add layer in the Attention layer.

[0135] Regarding the step of "using the processing unit in the computing unit to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data", the processing unit in the computing unit can be used to perform accumulation processing on the third output data and the fourth output data according to the network function of the accumulation processing functional layer in the feedforward neural network layer to obtain the output data.

[0136] Exemplarily, for a computing unit matching the Add layer in the FFN layer, the computing unit may utilize a processing unit to perform accumulation processing on each third output data and fourth output data according to an accumulation function, thereby obtaining output data of the computing unit.

[0137] In another embodiment, when the computing unit is a computing unit matched with a matrix-vector multiplication functional layer associated with a feature compression matrix, the input data includes the fifth output data sent by the computing unit matched with a matrix-vector multiplication functional layer associated with a feature mapping matrix, and the sixth output data sent by the computing unit matched with a matrix-vector multiplication functional layer associated with a feature change matrix.

[0138] Here, combined Figure 3It can be seen that for any computing unit matching the matmul(W2) layer, the fifth output data can be the output data of each computing unit matching the matmul(W1) layer; the sixth output data is the output data of each computing unit matching the matmul(W3) layer.

[0139] Exemplarily, the output data of each computing unit matching the matmul(W1) layer can be sent to each computing unit matching the matmul(W2) layer by broadcasting, and the output data of each computing unit matching the matmul(W3) layer can also be sent to each computing unit matching the matmul(W2) layer by broadcasting, and the fifth output data can be sent to each computing unit matching the matmul(W2) layer by broadcasting.

[0140] Furthermore, for the step of “using the processing unit in the computing unit to perform matrix-vector multiplication on the input data and the target data stored in the storage unit in the computing unit to obtain output data”, the processing unit in the computing unit can be used to perform matrix-vector multiplication on the fifth output data, the sixth output data, and the target data stored in the storage unit in the computing unit to obtain output data.

[0141] Exemplarily, for any computing unit matched with the matmul(W2) layer, the processing unit in the computing unit can be used to perform matrix-vector multiplication processing on each fifth output data, each sixth output data, and the target data stored in the storage unit of the computing unit to obtain output data. Specifically, each fifth output data can be processed with the target data by matrix-vector multiplication, and each sixth output data can be processed with the target data by matrix-vector multiplication to obtain output data related to each fifth output data and output data related to each sixth output data. Afterwards, each output data calculated by the computing unit can be sent to the Add layer in the FFN layer for accumulation processing to obtain the output data of the corresponding network layer.

[0142] In one embodiment, S102 may be implemented according to the following steps:

[0143] Determine a target functional layer that has a communication relationship with the functional layer matched with the computing unit; determine each other computing unit associated with the target functional layer; and send the output data to each other computing unit respectively by using the data sending interface included in the communication unit in the computing unit and the data receiving interface included in the communication unit in each other computing unit according to the communication connections and communication configuration files set for the communication units in different computing units.

[0144] Here, the target functional layer that has a communication relationship with any functional layer is the functional layer that the output data of the functional layer needs to input. For example, the target functional layer corresponding to the matmul(QKV) layer can be the multihead layer; the target functional layer corresponding to the Add layer in the Attention layer can be the Norm layer in the FFN layer and the Add layer in the FFN layer; the target functional layer corresponding to the Norm layer in the FFN layer can be the matmul(W1) layer and the matmul(W3) layer.

[0145] Optionally, the target functional layer may be determined based on at least one of a communication configuration file, a communication connection, and a processing order between functional layers.

[0146] After determining the target functional layer, the various computing units that match the target functional layer can be determined, and these computing units can be used as various other computing units that the current computing unit needs to communicate with. Then, according to the communication lines set for the communication units in various other computing units and the communication configuration files related to various other computing units, the data transmission interface included in the communication unit in the current computing unit, the data receiving interface included in the communication units in various other computing units, and the SPI connection between the current computing unit and various other computing units, the output data of the current computing unit is sent to various other computing units so that various other computing units can further process the data.

[0147] To facilitate understanding of the embodiments of the present disclosure, the following will be explained using the example of using a computing unit to implement reasoning of a single-layer target neural network. In this example, the computing unit is a chip, and the chip's storage unit SRAM = 32MB + 1MB. The parameters of the single-layer target neural network are as follows: hidden size hidden_size = 8192, intermediate size intermediate_size = 28672, head head = 64, key-value pair head nunkvheads = 8, and the corresponding parameter quantities of the single-layer target neural network are:

[0148] 8192*8192(Q)+8192*(64heads*16)*2(KV)+8192*8192(Attention)+8192*28672*3(FFN)=8192*8192*12.675=850M. It can be seen that the corresponding parameter amount of the single-layer target neural network is 850M parameters. Assuming that the parameters are stored in 8 bits, in the Attention layer, the memory required for each head (when calculating KVCache, assuming that the string length len=4096) is: 8192*128+8192*16*2+8192*128+len*16*2=2.5M. In the Attention layer, 12 heads can be placed on the same chip, so a total of 6 chips are required. In the input stage, the input data can be sent to the six chips through broadcasting; in the process of calculating the Q, K, and V matrices, row splitting can be used to calculate the final results of some positions; in the process of calculating the output, column splitting is used to calculate partial results of all positions, and then finally in the output, all the results of attention are output in the form of addition.

[0149] In the FFN layer, row splitting is used in both the w1 matrix and the w3 matrix to calculate the total value of the partial value; column splitting is used in the w2 matrix calculation to calculate the partial sum of all values ​​to obtain the final result of the FFN layer. For each chip, the calculation scale of 28672 is split into 23 parts, each of which is 1280 in size. In this way, the amount of data that needs to be stored in each chip is: 8192*1280+8192*1280+1280*8192=31M. In other words, each chip needs to store 31M of sub-matrix data. In this way, since the cost of chips is much lower than that of GPUs, using multiple chips instead of traditional GPUs for model inference can effectively reduce the cost of model inference.

[0150] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0151] Based on the same inventive concept, a data processing device corresponding to the data processing method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned data processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0152] like Figure 4 FIG. 1 is a schematic diagram of a data processing device provided by an embodiment of the present disclosure, comprising:

[0153] The processing module 401 is used for processing the input data using the processing unit in the computing unit that matches any network layer of the target neural network to obtain output data;

[0154] A sending module 402 is used to send the output data to at least one other computing unit by using the communication unit in the computing unit and the communication unit in other computing units according to the communication connection and communication configuration file set for the communication unit in different computing units; wherein the other computing unit includes other computing units matching the network layer, or computing units matching the next network layer;

[0155] The loop module 403 is used to return to the step of using the processing unit in the computing unit to process the input data to obtain output data, until the output data of the last computing unit matching the last network layer of the target neural network is obtained, and the output data is used as the data processing result.

[0156] In a possible implementation, the network layer includes multiple functional layers;

[0157] The multiple functional layers at least include a normalization processing functional layer located in the attention layer, a matrix-vector multiplication functional layer related to the query matrix, the key matrix and the value matrix, a multi-head functional layer, a matrix-vector multiplication functional layer related to the result output, an accumulation processing functional layer, and a normalization processing functional layer located in the feedforward neural network layer, a matrix-vector multiplication functional layer related to multiple weight matrices, and an accumulation processing functional layer;

[0158] A computing unit matching any network layer of the target neural network is associated with a functional layer among the multiple functional layers.

[0159] In a possible implementation manner, the processing module 401, when using the processing unit in the computing unit to process the input data to obtain the output data, is used to:

[0160] In the case where the computing unit is associated with any matrix-vector multiplication functional layer, using a processing unit in the computing unit, performing matrix-vector multiplication processing on the input data and target data stored in a storage unit in the computing unit to obtain output data;

[0161] Among them, the target data includes any preset matrix corresponding to the functional layer matching the computing unit or a sub-matrix obtained by matrix splitting any preset matrix, and the preset matrix includes a query matrix, a key matrix, a value matrix, a result output matrix and a weight matrix.

[0162] In a possible implementation manner, the processing module 401, when the processing unit in the computing unit is used to process the input data to obtain the output data, is used to:

[0163] In the case where the computing unit is independent of the matrix-vector multiplication functional layer, the processing unit in the computing unit is used to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data.

[0164] In a possible implementation, the device further includes:

[0165] The acquisition module 404, before using the processing unit in the computing unit to process the input data to obtain the output data, is used to:

[0166] In the case where the input data includes output data sent from multiple other computing units, the output data of each other computing unit is received and cached using multiple data receiving interfaces in the communication unit included in the computing unit;

[0167] In response to each of the data receiving interfaces having completed caching the corresponding output data, the calculation unit is used to trigger a reception completion signal to determine that the acquisition of the input data is completed.

[0168] In a possible implementation, the device further includes a detection module 405, configured to:

[0169] Performing functional testing and performance testing on the computing unit;

[0170] In case the detection fails, the computing unit is replaced by a spare computing unit.

[0171] In a possible implementation, the weight matrix includes a feature mapping matrix, a feature transformation matrix, and a feature compression matrix;

[0172] In the case where the preset matrix is ​​any one of the query matrix, the key matrix, the value matrix, the feature mapping matrix, and the feature transformation matrix, the submatrix included in the target data is a matrix obtained by performing row splitting on the preset matrix;

[0173] In the case where the preset matrix is ​​any one of a result output matrix and a feature compression matrix, the submatrix included in the target data is a matrix obtained by performing column splitting on the preset matrix.

[0174] In a possible implementation manner, if the preset matrix is ​​split into multiple sub-matrices, the multiple sub-matrices are respectively stored in storage units of different computing units;

[0175] The sizes of each sub-matrix corresponding to a preset matrix are related to at least one of the size of the preset matrix, the storage space of the storage unit in the calculation unit, and the computing capability of the processing unit in the calculation unit.

[0176] In a possible implementation manner, the processing module 401, when using the processing unit in the computing unit to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data, is used to:

[0177] In the case where the computing unit is a computing unit matching a multi-head functional layer in the attention layer, using a communication unit in the computing unit to obtain a matrix cache associated with a key matrix and a value matrix from a double rate DDR storage unit corresponding to the computing unit;

[0178] The processing unit is used to perform rotation position encoding processing on the input data and the matrix cache according to the network function of the multi-head functional layer to obtain output data.

[0179] In a possible implementation, when the computing unit is a computing unit matching an accumulation processing functional layer in an attention layer, the input data includes first output data sent by the last computing unit matching the previous network layer, and second output data sent by a computing unit matching a matrix-vector multiplication functional layer in the attention layer related to the result output;

[0180] The processing module 401, when using the processing unit in the computing unit to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data, is used to:

[0181] Utilizing the processing unit in the computing unit, the first output data and the second output data are accumulated and processed according to the network function of the accumulation processing function layer in the attention layer to obtain output data.

[0182] In a possible implementation, when the computing unit is a computing unit matching an accumulation processing functional layer in a feedforward neural network layer, the input data includes third output data sent by a computing unit matching a matrix-vector multiplication functional layer associated with a feature compression matrix, and fourth output data sent by a computing unit matching an accumulation processing functional layer in an attention layer;

[0183] The processing module 401, when using the processing unit in the computing unit to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data, is used to:

[0184] The processing unit in the computing unit is used to perform accumulation processing on the third output data and the fourth output data according to the network function of the accumulation processing function layer in the feedforward neural network layer to obtain output data.

[0185] In a possible implementation manner, when the computing unit is a computing unit matched with a matrix-vector multiplication function layer associated with a feature compression matrix, the input data includes fifth output data sent by a computing unit matched with a matrix-vector multiplication function layer associated with a feature mapping matrix, and sixth output data sent by a computing unit matched with a matrix-vector multiplication function layer associated with a feature change matrix;

[0186] The processing module 401, when using the processing unit in the computing unit to perform matrix-vector multiplication processing on the input data and the target data stored in the storage unit in the computing unit to obtain output data, is used to:

[0187] The processing unit in the computing unit is used to perform matrix-vector multiplication processing on the fifth output data, the sixth output data and the target data stored in the storage unit in the computing unit to obtain output data.

[0188] In a possible implementation manner, the sending module 402, when sending the output data to at least one other computing unit using the communication unit in the computing unit and the communication unit in other computing units according to the communication connection and communication configuration file set for the communication units in different computing units, is used to:

[0189] Determine a target functional layer having a communication relationship with the functional layer matched with the computing unit;

[0190] Determining each other computing unit associated with the target functional layer;

[0191] According to the communication connections and communication configuration files set for the communication units in different computing units, the output data is sent to each other computing unit respectively by utilizing the data sending interface included in the communication unit in the computing unit and the data receiving interface included in the communication units in each other computing unit.

[0192] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0193] Based on the same technical concept, the embodiment of the present application also provides a computer device. Figure 5 FIG. 1 is a schematic diagram of a computer device provided in an embodiment of the present application, including:

[0194] Processor 501, memory 502 and bus 503. The memory 502 stores machine-readable instructions executable by the processor 501, and the processor 501 is used to execute the machine-readable instructions stored in the memory 502. When the machine-readable instructions are executed by the processor 501, the processor 501 performs the following steps: S101: for a computing unit matching any network layer of a target neural network, use a processing unit in the computing unit to process input data to obtain output data; S102: according to the communication lines and communication configuration files set for the communication units in different computing units, use the communication unit in the computing unit and the communication unit in other computing units to send the output data to at least one other computing unit; wherein the other computing units include other computing units matching the network layer, or computing units matching the next network layer; and S103: return to the step of using the processing unit in the computing unit to process the input data to obtain output data, until the output data of the last computing unit matching the last network layer of the target neural network is obtained, and the output data is used as the data processing result.

[0195] The above-mentioned memory 502 includes a memory 5021 and an external memory 5022; the memory 5021 here is also called an internal memory, which is used to temporarily store the calculation data in the processor 501, as well as the data exchanged with the external memory 5022 such as a hard disk. The processor 501 exchanges data with the external memory 5022 through the memory 5021. When the computer device is running, the processor 501 communicates with the memory 502 through the bus 503, so that the processor 501 executes the execution instructions mentioned in the above method embodiment.

[0196] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data processing method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0197] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the software update method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0198] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0199] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0200] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0201] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0202] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0203] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0204] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the aforementioned embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: include: For a computing unit that matches any network layer of the target neural network, using a processing unit in the computing unit, performing data processing on input data to obtain output data; According to the communication lines and communication configuration files set for the communication units in different computing units, the output data is sent to at least one other computing unit by using the communication unit in the computing unit and the communication units in other computing units; wherein the other computing units include other computing units matching the network layer, or computing units matching the next network layer; Return to the step of using the processing unit in the computing unit to process the input data to obtain output data, until the output data of the last computing unit matching the last network layer of the target neural network is obtained, and the output data is used as the data processing result.

2. The method according to claim 1, characterized in that The network layer includes multiple functional layers; The multiple functional layers at least include a normalization processing functional layer located in the attention layer, a matrix-vector multiplication functional layer related to the query matrix, the key matrix and the value matrix, a multi-head functional layer, a matrix-vector multiplication functional layer related to the result output, an accumulation processing functional layer, and a normalization processing functional layer located in the feedforward neural network layer, a matrix-vector multiplication functional layer related to multiple weight matrices, and an accumulation processing functional layer; A computing unit matching any network layer of the target neural network is associated with a functional layer among the multiple functional layers.

3. The method according to claim 1, characterized in that The step of using the processing unit in the computing unit to process the input data to obtain output data includes: In the case where the computing unit is associated with any matrix-vector multiplication functional layer, using a processing unit in the computing unit, performing matrix-vector multiplication processing on the input data and target data stored in a storage unit in the computing unit to obtain output data; Among them, the target data includes any preset matrix corresponding to the functional layer matching the computing unit or a sub-matrix obtained by matrix splitting any preset matrix, and the preset matrix includes a query matrix, a key matrix, a value matrix, a result output matrix and a weight matrix.

4. The method according to claim 1, characterized in that: The processing unit in the computing unit processes the input data to obtain output data, further comprising: In the case where the computing unit is independent of the matrix-vector multiplication functional layer, the processing unit in the computing unit is used to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data.

5. The method according to claim 1, characterized in that Before using the processing unit in the computing unit to process the input data to obtain the output data, the method further includes: In the case where the input data includes output data sent from multiple other computing units, the output data of each other computing unit is received and cached using multiple data receiving interfaces in the communication unit included in the computing unit; In response to each of the data receiving interfaces having completed caching the corresponding output data, the calculation unit is used to trigger a reception completion signal to determine that the acquisition of the input data is completed.

6. The method according to claim 1, characterized in that The method further comprises: Performing functional testing and performance testing on the computing unit; In case the detection fails, the computing unit is replaced by a spare computing unit.

7. The method according to claim 3, characterized in that The weight matrix includes a feature mapping matrix, a feature transformation matrix and a feature compression matrix; In the case where the preset matrix is ​​any one of the query matrix, the key matrix, the value matrix, the feature mapping matrix, and the feature transformation matrix, the submatrix included in the target data is a matrix obtained by performing row splitting on the preset matrix; In the case that the preset matrix is ​​any one of a result output matrix and a feature compression matrix, the submatrix included in the target data is a matrix obtained by performing column splitting on the preset matrix.

8. The method according to claim 7, characterized in that If the preset matrix is ​​split into multiple sub-matrices, the multiple sub-matrices are respectively stored in storage units of different computing units; The sizes of each sub-matrix corresponding to a preset matrix are related to at least one of the size of the preset matrix, the storage space of the storage unit in the calculation unit, and the computing capability of the processing unit in the calculation unit.

9. The method according to claim 4, characterized in that The using the processing unit in the computing unit to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data includes: In the case where the computing unit is a computing unit matching a multi-head functional layer in the attention layer, using a communication unit in the computing unit to obtain a matrix cache associated with a key matrix and a value matrix from a double rate DDR storage unit corresponding to the computing unit; The processing unit is used to perform rotation position encoding processing on the input data and the matrix cache according to the network function of the multi-head functional layer to obtain output data.

10. The method according to claim 4, characterized in that In the case where the computing unit is a computing unit that matches an accumulation processing functional layer located in the attention layer, the input data includes the first output data sent by the last computing unit that matches the previous network layer, and the second output data sent by the computing unit that matches the matrix-vector multiplication functional layer related to the result output located in the attention layer; The using the processing unit in the computing unit to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data includes: Utilizing the processing unit in the computing unit, the first output data and the second output data are accumulated and processed according to the network function of the accumulation processing function layer in the attention layer to obtain output data.

11. The method according to claim 4, characterized in that In the case where the computing unit is a computing unit matched with an accumulation processing functional layer located in a feedforward neural network layer, the input data includes third output data sent by a computing unit matched with a matrix-vector multiplication functional layer associated with a feature compression matrix, and fourth output data sent by a computing unit matched with an accumulation processing functional layer located in an attention layer; The using the processing unit in the computing unit to perform corresponding functional processing on the input data according to the network function of the functional layer related to the computing unit to obtain output data includes: The processing unit in the computing unit is used to perform accumulation processing on the third output data and the fourth output data according to the network function of the accumulation processing function layer in the feedforward neural network layer to obtain output data.

12. The method according to claim 3, characterized in that In the case where the computing unit is a computing unit matched with a matrix-vector multiplication function layer associated with a feature compression matrix, the input data includes fifth output data sent by a computing unit matched with a matrix-vector multiplication function layer associated with a feature mapping matrix, and sixth output data sent by a computing unit matched with a matrix-vector multiplication function layer associated with a feature change matrix; The method of using the processing unit in the computing unit to perform matrix-vector multiplication processing on the input data and the target data stored in the storage unit in the computing unit to obtain output data includes: The processing unit in the computing unit is used to perform matrix-vector multiplication processing on the fifth output data, the sixth output data and the target data stored in the storage unit in the computing unit to obtain output data.

13. The method according to claim 1, characterized in that The method of sending the output data to at least one other computing unit by using the communication unit in the computing unit and the communication units in other computing units according to the communication lines and communication configuration files set for the communication units in different computing units includes: Determine a target functional layer having a communication relationship with the functional layer matched with the computing unit; Determining each other computing unit associated with the target functional layer; According to the communication connections and communication configuration files set for the communication units in different computing units, the output data is sent to each other computing unit respectively by utilizing the data sending interface included in the communication unit in the computing unit and the data receiving interface included in the communication units in each other computing unit.

14. A data processing device, characterized in that: include: A processing module, for processing input data using a processing unit in a computing unit that matches any network layer of a target neural network, to obtain output data; A sending module, configured to send the output data to at least one other computing unit by using the communication unit in the computing unit and the communication unit in other computing units according to the communication connection and communication configuration file set for the communication units in different computing units; wherein the other computing units include other computing units matching the network layer, or computing units matching the next network layer; The loop module is used to return to the step of using the processing unit in the computing unit to process the input data to obtain output data, until the output data of the last computing unit matching the last network layer of the target neural network is obtained, and the output data is used as the data processing result.

15. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor performs the steps of the data processing method according to any one of claims 1 to 13.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a computer device, the computer device executes the steps of the data processing method according to any one of claims 1 to 13.

Images