A data processing method, device and equipment

Abstract

The present application provides a data processing method, device and equipment, the method comprising: if the input feature of the target unit operator corresponds to multiple dimensions, then select the dimension to be segmented from the multiple dimensions, segment the input feature on the dimension to be segmented to obtain multiple segmentation sub-data; traverse the first segmentation sub-data and store it in the first internal storage medium; read the first segmentation sub-data through the first calculation unit, process based on the first segmentation sub-data to obtain the first result sub-data, and store the first result sub-data in the second internal storage medium; read the first result sub-data through the second calculation unit, process based on the first result sub-data to obtain the second result sub-data; traverse the second segmentation sub-data and store it in the first internal storage medium, read the second segmentation sub-data through the first calculation unit, and process based on the second segmentation sub-data to obtain the first result sub-data. Through the scheme of the present application, the memory access performance is improved and the bandwidth utilization is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device and equipment. Background Art

[0002] In the field of artificial intelligence, after training a network model, the network model can be deployed on a hardware device, and the data to be detected (such as image data, video data, or audio data, etc.) can be inferred based on the network model to obtain the detection results of the data to be detected. However, when the hardware device infers the data to be detected based on the network model, it brings huge computing overhead and low memory utilization.

[0003] Take the Transformer network model as an example. The Transformer network model is a deep learning model that is mainly used to process sequence data. The Transformer network model introduces a self-attention mechanism to solve the gradient vanishing problem encountered when processing long sequences. However, during the operation of the Transformer network model, a large amount of data needs to be copied to the internal storage medium of the processor. When a large amount of data is placed in the internal storage medium of the processor, the utilization rate of the internal storage medium will be reduced, the memory access performance will be seriously reduced, and even the Transformer network model will not be able to run normally. In addition, the processor needs to perform a large number of matrix operations and vector operations, which will take up a lot of computing time of the processor, resulting in huge computing overhead. Summary of the invention

[0004] The present application provides a data processing method, which is applied to a processor, wherein the processor deploys a target network model, wherein the target network model includes a target unit operator, wherein the target unit operator includes a first computing unit and a second computing unit, wherein the first computing unit corresponds to a first internal storage medium of the processor, and the second computing unit corresponds to a second internal storage medium of the processor, wherein the method includes:

[0005] If the input feature of the target unit operator corresponds to multiple dimensions, a dimension to be segmented is selected from the multiple dimensions, and the input feature is segmented on the dimension to be segmented to obtain multiple segmented sub-data; wherein the input feature is stored in an external storage medium;

[0006] Traversing the first cutting sub-data and storing them in a first internal storage medium; reading the first cutting sub-data by a first computing unit, processing the first cutting sub-data to obtain first result sub-data, and storing the first result sub-data in a second internal storage medium; reading the first result sub-data by a second computing unit, processing the first result sub-data to obtain second result sub-data;

[0007] After reading the first segmentation sub-data by the first computing unit, traversing the second segmentation sub-data and storing them in the first internal storage medium, returning to execute the step of reading the second segmentation sub-data by the first computing unit, and processing the second segmentation sub-data to obtain the first result sub-data;

[0008] The process of processing based on the second cutting sub-data by the first computing unit and the process of processing based on the first result sub-data by the second computing unit are executed in parallel.

[0009] The present application provides a data processing device, which is applied to a processor, wherein the processor deploys a target network model, wherein the target network model includes a target unit operator, wherein the target unit operator includes a first computing unit and a second computing unit, wherein the first computing unit corresponds to a first internal storage medium of the processor, and the second computing unit corresponds to a second internal storage medium of the processor, wherein the device includes:

[0010] A selection module, configured to select a dimension to be segmented from the multiple dimensions if the input feature of the target unit operator corresponds to multiple dimensions, and the input feature is stored in an external storage medium;

[0011] A processing module is used to segment the input feature on the to-be-segmented dimension to obtain a plurality of segmented sub-data; traverse the first segmented sub-data and store them in a first internal storage medium;

[0012] A control module, configured to read first segmentation sub-data from a first internal storage medium through a first computing unit, obtain first result sub-data based on the first segmentation sub-data by processing, and store the first result sub-data in a second internal storage medium; read the first result sub-data from a second internal storage medium through a second computing unit, obtain second result sub-data based on the first result sub-data by processing;

[0013] The processing module is further configured to traverse the second segmentation sub-data and store them in the first internal storage medium after the control module reads the first segmentation sub-data from the first internal storage medium through the first calculation unit;

[0014] The control module is further configured to read the second cutting sub-data from the first internal storage medium through the first computing unit, and obtain the first result sub-data by processing based on the second cutting sub-data;

[0015] The process of processing based on the second cutting sub-data by the first computing unit and the process of processing based on the first result sub-data by the second computing unit are executed in parallel.

[0016] The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the data processing method of the above example of the present application.

[0017] The present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the data processing method of the above example of the present application is implemented.

[0018] The present application provides a machine-readable storage medium, which stores machine-executable instructions that can be executed by a processor; wherein the processor is used to execute the machine-executable instructions to implement the data processing method of the above example of the present application.

[0019] It can be seen from the above technical solutions that in the embodiment of the present application, during the operation of the target network model, a plurality of segmented sub-data are obtained by segmenting the input features, and the segmented sub-data are stored in the internal storage medium of the processor, so that only a small amount of data is placed in the internal storage medium of the processor, thereby improving the utilization rate of the internal storage medium of the processor and improving the memory access performance. For different segmented sub-data (first segmented sub-data and second segmented sub-data), the processor can perform the operation of the first segmented sub-data and the second segmented sub-data in parallel, saving the calculation time of the processor and the calculation overhead of the processor. When the hardware device infers the input data based on the network model, it saves computing resources and only requires fewer computing resources to complete data inference. It can realize the parallelization of calculation and transport pipelines to the greatest extent, taking into account the time consumption of inference and the bandwidth occupancy rate, and maximize the utilization rate of the hardware device as much as possible. By accelerating the first computing unit and the second computing unit in parallel, it is possible to optimize the bandwidth while ensuring the execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of a data processing method in one embodiment of the present application;

[0021] Figure 2 It is a schematic diagram of a process of deploying a target network model in one embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of a unified graphical representation of a model to be deployed in one embodiment of the present application;

[0023] Figure 4 is a schematic diagram of graph compilation in one embodiment of the present application;

[0024] Figure 5A It is a flowchart of a data processing method in one embodiment of the present application;

[0025] Figure 5B It is a schematic diagram of selecting the dimension to be segmented in one embodiment of the present application;

[0026] Figure 5C is a schematic diagram of an internal storage medium and an external storage medium in one embodiment of the present application;

[0027] Figure 5D is a schematic diagram of a processor in one embodiment of the present application enabling multiple pipelines at the same time;

[0028] Figure 5E It is a flowchart of a data processing method in one embodiment of the present application;

[0029] Fig. 5F is a schematic diagram of a processor in one embodiment of the present application enabling multiple pipelines at the same time;

[0030] Fig. 6A is a schematic diagram of the structure of a self-attention unit operator in one embodiment of the present application;

[0031] Figure 6B It is a flowchart of a data processing method in one embodiment of the present application;

[0032] Figure 6C is a processing schematic diagram of a self-attention unit operator in one embodiment of the present application;

[0033] Fig.6D is a schematic diagram of concurrent execution of Softmax and MatMul in one implementation of the present application;

[0034] Figure 7 is a schematic diagram of the structure of a data processing device in one embodiment of the present application;

[0035] Figure 8 It is a hardware structure diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0036] A data processing method is proposed in an embodiment of the present application. The method can be applied to a processor. The processor deploys a target network model. The input data of the target network model can be image data, video data or voice data. That is, the target network model is used to process the image data, video data or voice data to obtain a data processing result. For example, when the target network model is used to implement a detection function, the data processing result is a detection result. When the target network model is used to implement a classification function, the data processing result is a classification result.

[0037] Exemplarily, the target network model may include a target unit operator, which may include a first computing unit and a second computing unit, the first computing unit corresponds to a first internal storage medium located in a processor, and the second computing unit corresponds to a second internal storage medium located in the processor.

[0038] See also Figure 1 FIG. 4 is a flow chart of the method, which may include:

[0039] Step 101: If the input feature of the target unit operator corresponds to multiple dimensions, a dimension to be segmented is selected from the multiple dimensions, and the input feature is segmented on the dimension to be segmented to obtain multiple segmented sub-data; wherein the input feature is stored in an external storage medium.

[0040] Step 102, traverse the first cutting sub-data and store them in the first internal storage medium; read the first cutting sub-data through the first computing unit, process the first cutting sub-data to obtain first result sub-data, and store the first result sub-data in the second internal storage medium; read the first result sub-data through the second computing unit, and process the first result sub-data to obtain second result sub-data.

[0041] Step 103: after reading the first slice sub-data through the first computing unit, traverse the second slice sub-data and store them in the first internal storage medium, and return to execute the step of reading the second slice sub-data through the first computing unit and processing based on the second slice sub-data to obtain the first result sub-data.

[0042] Exemplarily, a process of processing based on the second cut sub-data by the first computing unit and a process of processing based on the first result sub-data by the second computing unit are executed in parallel.

[0043] Exemplarily, selecting the dimension to be split from multiple dimensions may include, but is not limited to: selecting the first dimension with the largest data volume from multiple dimensions. If the data volume of the first dimension is greater than the product of the data volumes of all dimensions in the dimension set, the first dimension may be determined as the dimension to be split; wherein the dimension set may include at least two dimensions, and the dimension set includes all dimensions or part of the dimensions except the first dimension. If the data volume of the first dimension is not greater than the product of the data volumes of all dimensions in the dimension set, and the product of the data volume of the first dimension and the data volume of all dimensions in the dimension set is greater than the upper limit data volume of the first internal storage medium, all dimensions in the dimension set may be determined as the dimensions to be split. If the data volume of the first dimension is not greater than the product of the data volumes of all dimensions in the dimension set, and the product of the data volume of the first dimension and the data volume of all dimensions in the dimension set is not greater than the upper limit data volume, the first dimension and all dimensions in the dimension set may be determined as the dimensions to be split.

[0044] Exemplarily, after the first result sub-data is read by the second computing unit and the second result sub-data is obtained by processing based on the first result sub-data, the second result sub-data can also be stored in an external storage medium; wherein the second result sub-data corresponding to all the cut sub-data are used to generate the output characteristics of the target unit operator. Alternatively, the second result sub-data is stored in a third internal storage medium by the second computing unit, wherein the first computing unit can also correspond to the third internal storage medium located in the processor; the second result sub-data is read by the first computing unit, and the third result sub-data is obtained by processing based on the second result sub-data; wherein the process of processing based on the second cut sub-data by the first computing unit is executed in parallel with the process of processing based on the second result sub-data by the first computing unit. Then, the third result sub-data can be stored in an external storage medium; the third result sub-data corresponding to all the cut sub-data are used to generate the output characteristics of the target unit operator.

[0045] Exemplarily, the first computing unit may include a matrix computing unit, and the second computing unit may include a non-matrix computing unit; or, the first computing unit may include a non-matrix computing unit, and the second computing unit may include a matrix computing unit; wherein the matrix computing unit may include a unit for performing matrix operations; and the non-matrix computing unit may include a unit for performing non-matrix operations.

[0046] Exemplarily, if the target unit operator includes a self-attention unit operator, and the self-attention unit operator can include a first MatMul process, a Scale process, a Mask process, a Softmax process, and a second MatMul process. Based on this, the first slice sub-data can be read from the first internal storage medium by the first computing unit, and the first MatMul process, the Scale process, and the Mask process can be performed based on the first slice sub-data to obtain the first result sub-data, and the first result sub-data can be stored in the second internal storage medium; the first result sub-data can be read from the second internal storage medium by the second computing unit, and the Softmax process can be performed based on the first result sub-data to obtain the second result sub-data, and the second result sub-data can be stored in the third internal storage medium; the second result sub-data can be read from the third internal storage medium by the first computing unit, and the second MatMul process can be performed based on the second result sub-data to obtain the third result sub-data.

[0047] Exemplarily, the input features of the self-attention unit operator may include a query vector, a keyword vector, and a value vector, wherein the query vector may have a length of K and a height of M, the keyword vector may have a length of N and a height of K, and the value vector may have a length of K and a height of N. The multiple dimensions corresponding to the input features may include a K dimension, an M dimension, and an N dimension.

[0048] Among them, the first dimension is K dimension, and all dimensions in the dimension set are M dimensions and N dimensions; or, the first dimension is M dimension, and all dimensions in the dimension set are K dimensions and N dimensions; or, the first dimension is N dimension, and all dimensions in the dimension set are K dimensions and M dimensions.

[0049] Exemplarily, in the process of splitting the input features on the dimension to be split to obtain multiple split sub-data, if the dimension to be split includes both M dimension and N dimension, the absolute value of the difference between the split value m of the M dimension and the split value n of the N dimension is less than the first threshold; or, if the dimension to be split includes the M dimension but does not include the N dimension, the absolute value of the difference between the split value m of the M dimension and the specification value of the N dimension is less than the second threshold; or, if the dimension to be split includes the N dimension but does not include the M dimension, the absolute value of the difference between the split value n of the N dimension and the specification value of the M dimension is less than the third threshold.

[0050] It can be seen from the above technical solutions that in the embodiment of the present application, during the operation of the target network model, a plurality of segmented sub-data are obtained by segmenting the input features, and the segmented sub-data are stored in the internal storage medium of the processor, so that only a small amount of data is placed in the internal storage medium of the processor, thereby improving the utilization rate of the internal storage medium of the processor and improving the memory access performance. For different segmented sub-data (first segmented sub-data and second segmented sub-data), the processor can perform the operation of the first segmented sub-data and the second segmented sub-data in parallel, saving the calculation time of the processor and the calculation overhead of the processor. When the hardware device infers the input data based on the network model, it saves computing resources and only requires fewer computing resources to complete data inference. It can realize the parallelization of calculation and transport pipelines to the greatest extent, taking into account the time consumption of inference and the bandwidth occupancy rate, and maximize the utilization rate of the hardware device as much as possible. By accelerating the first computing unit and the second computing unit in parallel, it is possible to optimize the bandwidth while ensuring the execution efficiency.

[0051] The technical solution of the embodiment of the present application is described below in conjunction with specific application scenarios.

[0052] In the field of artificial intelligence, after training a network model, the network model is deployed on a hardware device, and the data (such as image data, video data, or audio data) is inferred based on the network model to obtain the inference result. When inferring data based on the network model, it brings huge computing overhead and low memory utilization.

[0053] In response to the above findings, a data processing method is proposed in an embodiment of the present application, involving multi-dimensional fields such as video acquisition and analysis, natural language, etc., by segmenting the data to obtain multiple segmented sub-data, and storing the segmented sub-data in the internal storage medium of the processor, so that only a small amount of data is placed in the internal storage medium of the processor, thereby improving the utilization rate of the internal storage medium of the processor and improving memory access performance.

[0054] Exemplarily, in order to implement the data processing method in the embodiment of the present application, it is necessary to deploy the target network model on the processor of the hardware device. The deployment process of the target network model is described below. Figure 2 FIG. 1 is a flow chart of deploying a target network model. The process may include:

[0055] Step 201: Obtain a model to be deployed. The model to be deployed may be a trained network model or any type of network model, such as a deep learning model. There is no restriction on the type of the model to be deployed.

[0056] Step 202: Perform a unified graph representation on the model to be deployed. For example, the model to be deployed may be converted into a unified representation (Intermediate Representation, IR for short), that is, a unified graph representation.

[0057] For example, see Figure 3 As shown in the figure, it is a schematic diagram of a unified graph representation of the model to be deployed. The model to be deployed can be a network model trained by the front-end platform, and the front-end platform may be a Caffe front-end platform, an ONNC front-end platform, a TensorFlow front-end platform, or other front-end platforms, and the hardware device to be deployed by the model to be deployed may be a hardware device of type A, a hardware device of type B, a hardware device of type C, or other types of hardware devices. Based on this, in order to enable various types of hardware devices to directly process the network model, a unified graph representation can be performed on the model to be deployed, and the models to be deployed trained by various front-end platforms can be parsed and converted into a unified representation (IR), so that various types of hardware devices use the information in the unified representation when constructing operators, and are unaware of the type of the front-end platform.

[0058] The unified representation (IR) can be a unified and universal data structure used to characterize the features of the model to be deployed. In order to convert the model to be deployed into the unified representation, the model to be deployed can be converted into the unified representation using MLIR, or using MindIR, or using a universal data structure. No restrictions are imposed on this. Regardless of the conversion method used, the unified representation needs to contain basic information of the operator, such as layer information (such as weights, biases, inputs, outputs, etc.) and layer attributes (such as axes, algorithm types, etc.).

[0059] Step 203: Optimize the graph based on the unified graph representation to obtain an optimized subgraph.

[0060] Exemplarily, a graph optimization can be performed based on a unified graph representation to obtain an optimized subgraph. The unified graph representation includes basic information of multiple operators. For each operator, the graph optimization of the operator can be performed based on the basic information of the operator. In order to optimize the graph of the operator, optimization methods such as operator graph fusion, operator graph parallelism, and operator graph segmentation can be used. There is no restriction on this optimization process, as long as the operator can be optimized. Of course, for some operators, the graph optimization of the operator may not be performed, and it can be kept unchanged.

[0061] In this embodiment, all operators can be divided into target unit operators and non-target unit operators, where the target unit operators are operators including the first computing unit and the second computing unit, and the non-target unit operators are operators other than the target unit operators. For example, the first computing unit may include a matrix computing unit, and the second computing unit may include a non-matrix computing unit; or, the first computing unit may include a non-matrix computing unit, and the second computing unit may include a matrix computing unit. The matrix computing unit includes a unit for performing matrix operations, and the non-matrix computing unit includes a unit for performing non-matrix operations.

[0062] From the above, it can be seen that an operator that includes both matrix calculation units and non-matrix calculation units can be called a target unit operator, an operator that includes only matrix calculation units can be called a non-target unit operator, and an operator that includes only non-matrix calculation units can be called a non-target unit operator.

[0063] For matrix computing units, the execution efficiency of matrix computing units is high, but there are more constraints. For example, the amount of data that can be calculated in one computing cycle is limited. The reasoning performance of matrix computing units is the key to the deployment of the entire network. For non-matrix computing units, non-matrix computing units can perform vector operations, and non-matrix computing units can also perform scalar operations, such as program loop control, branch judgment, etc.

[0064] For non-target unit operators, optimization methods such as operator graph fusion, operator graph parallelism, and operator graph segmentation can be used to optimize the graph of non-target unit operators. There is no restriction on this optimization process.

[0065] For the target unit operator, optimization methods such as operator graph fusion, operator graph parallelism, and operator graph segmentation can be used to optimize the graph of the target unit operator, and there is no restriction on this optimization process.

[0066] On this basis, the graph optimization method of this embodiment can also be used to perform graph optimization on the target unit operator. Based on the graph optimization, the input features of the target unit operator can be segmented to obtain multiple segmented sub-data, and the segmented sub-data are stored in the internal storage medium of the processor (rather than storing the input features in the internal storage medium of the processor). For different segmented sub-data, operations on different segmented sub-data are performed in parallel, and parallel acceleration is achieved for the first computing unit and the second computing unit.

[0067] Exemplarily, performing graph optimization on a target unit operator actually configures a processing rule (also referred to as a processing strategy) for the inference library of the target unit operator, so that during the inference process of the model, the inference library of the target unit operator uses the processing rule to process the input features of the target unit operator.

[0068] For the graph optimization process, it is a process of configuring processing rules for the inference library of the target unit operator. There is no restriction on this process, as long as the inference library can realize the function of "segmenting the input features of the target unit operator to obtain multiple segmented sub-data based on the processing rules, storing the segmented sub-data to the internal storage medium of the processor, executing operations on different segmented sub-data in parallel for different segmented sub-data, and realizing parallel acceleration for the first computing unit and the second computing unit". The implementation process of this function is actually the data processing method of this embodiment. The data processing method refers to the subsequent embodiments.

[0069] Step 204: compile the optimized subgraph to obtain a target network model.

[0070] For example, see Figure 4 As shown in the figure, it is a schematic diagram of graph compilation. The optimized subgraph represented by the unified graph can be input into the AI ​​compiler, and the optimized subgraph is compiled by the AI ​​compiler (i.e., graph compilation) to obtain the target network model. For example, the calculation logic is extracted from the optimized subgraph, and the calculation logic is compiled as the input of the AI ​​compiler to generate the target network model. There is no restriction on this compilation process, as long as the optimized subgraph can be compiled into the target network model. For example, the AI ​​compiler can be a general AI compiler or a third-party AI compiler, and there is no restriction on the selection of this AI compiler.

[0071] Exemplarily, steps 201 to 204 may be implemented by an offline optimization device, which may be a server, a management device, a control device, etc. The offline optimization device may also be an offline mirror, and there is no limitation on this. After obtaining the target network model, the offline optimization device may deploy the target network model to a hardware device, such as a hardware device of type A, a hardware device of type B, etc.

[0072] Exemplarily, after obtaining the target network model, the target network model can be deployed on the processor of the hardware device, that is, the processor runs the target network model to detect the data to be detected and obtain the detection result of the data to be detected. The data to be detected can be image data, video data or audio data, and there is no restriction on the type of the data to be detected. The processor can be a CPU (central processing unit), a GPU (graphics processing unit) or an NPU (network processing unit), and there is no restriction on the type of the processor. The hardware device can be an IPC (network camera), an industry camera, etc., and there is no restriction on the type of the hardware device.

[0073] For example, when the processor runs the target network model, it can implement a perimeter algorithm, that is, define a detection area, and detect whether the object in the image is within the area. Of course, the above is only an example of the function of the target network model, and the function of the target network model is not limited in this embodiment.

[0074] A data processing method is proposed in an embodiment of the present application. The method can be applied to a processor. The processor can deploy a target network model, and the target network model can include a target unit operator.

[0075] For the target network model, the target unit operator may include other operators (represented as operator A1) in front of it, or may not include other operators. If the target unit operator includes operator A1 (composed of one or more operators) in front of it, assuming that the input data of the target network model is the data to be detected, the data to be detected is processed by operator A1 to obtain the processed data (i.e., the output data of operator A1), and the output data of operator A1 is used as the input feature of the target unit operator. Alternatively, if the target unit operator does not include other operators in front of it, assuming that the input data of the target network model is the data to be detected, the data to be detected is used as the input feature of the target unit operator. Based on this, the input feature can be processed by the target unit operator to obtain the processed data, i.e., the output feature of the target unit operator.

[0076] For the target network model, the target unit operator may include other operators (represented as operator A2) or may not include other operators. If the target unit operator includes operator A2 (composed of one or more operators), the input data of operator A2 is the output feature of the target unit operator, and the output feature is processed by operator A2 to obtain the processed data (i.e., the output data of operator A2), and the output data of operator A2 is used as the output data of the target network model, and then the detection result corresponding to the data to be detected is determined based on the output data. Alternatively, if the target unit operator does not include other operators, the output feature of the target unit operator is used as the output data of the target network model, and then the detection result corresponding to the data to be detected is determined based on the output data.

[0077] The number of target unit operators may be one or more. If there are more than one target unit operators, each target unit operator may be processed in the above manner. For the convenience of description, one target unit operator will be used as an example for explanation.

[0078] The processing process of the non-target unit operator (such as operator A1 and / or operator A2) of the target network model is not limited in this embodiment. The data processing method is introduced later by taking the processing process of the target unit operator of the target network model as an example. It should be noted that in the processing process of the target unit operator of the target network model, in the inference process of the target unit operator, the inference library of the target unit operator uses the processing rules to process the input features of the target unit operator to obtain the output features of the target unit operator, that is, the data processing method is implemented by the inference library of the target unit operator.

[0079] In the embodiment of the present application, a data processing method is proposed, which can be applied to a processor, see Figure 5A FIG. 1 is a flow chart of the data processing method, which may include:

[0080] Step 511: If the input feature of the target unit operator corresponds to multiple dimensions, a dimension to be segmented is selected from the multiple dimensions (the dimension to be segmented may be one dimension or at least two dimensions), and the input feature is segmented on the dimension to be segmented to obtain multiple segmented sub-data.

[0081] Exemplarily, if the input feature corresponds to multiple dimensions, you can arbitrarily select one dimension as the dimension to be split, you can select the dimension with the largest data volume as the dimension to be split, you can arbitrarily select two dimensions as the dimensions to be split, you can select the two dimensions with the largest data volume and the second largest data volume as the dimensions to be split, and there is no restriction on the method of selecting the dimensions to be split.

[0082] For example, see Figure 5B As shown, the dimension to be segmented can be selected in the following way:

[0083] Step 521: Obtain multiple dimensions of input features of the target unit operator.

[0084] Taking the three dimensions of the input feature as an example, the three dimensions can be C dimension, H dimension and W dimension. The C dimension represents the number of channels of the input feature (that is, how many channels there are in the channel direction), the H dimension represents the height of the input feature (that is, how many pixels there are in the height direction), and the W dimension represents the width of the input feature (that is, how many pixels there are in the width direction). Of course, the above are just examples of multiple dimensions of the input feature, and there is no restriction on the dimensions of the input feature. Of course, the C dimension, H dimension and W dimension are just examples, and there is no restriction on this. The input feature can also have more dimensions, such as N dimension, etc., where N represents the number dimension.

[0085] Step 522: determine whether the data volume of the first dimension is greater than the product of the data volumes of all dimensions in the dimension set. If so, execute step 523; if not, execute step 524.

[0086] Exemplarily, the dimension with the largest amount of data can be selected from multiple dimensions as the first dimension, and a dimension set can be constructed based on the remaining dimensions except the first dimension. The dimension set can include at least two dimensions, and the dimension set includes all dimensions or some dimensions except the first dimension. For example, assuming that the amount of data in the C dimension is greater than the amount of data in the H dimension (that is, the number of channels in the C dimension is greater than the number of pixels in the H dimension), and the amount of data in the C dimension is greater than the amount of data in the W dimension (that is, the number of channels in the C dimension is greater than the number of pixels in the W dimension), then the first dimension is the C dimension, and the dimension set includes the H dimension and the W dimension.

[0087] Based on this, in step 522, see Figure 5B As shown, it is necessary to determine whether C is greater than H*W. If so, step 523 can be executed, and if not, step 524 can be executed.

[0088] Step 523: determine the first dimension as the dimension to be segmented. For example, the C dimension can be determined as the dimension to be segmented, that is, segmentation is performed in the C dimension, that is, the input features are segmented in the C dimension.

[0089] Exemplarily, dividing the input feature in the C dimension to obtain multiple segmented sub-data means that, assuming that the input feature is a C*H*W feature, the input feature can be divided into c segmented sub-data. For example, the first segmented sub-data is a 1*H*W feature, corresponding to a feature of C=1, the second segmented sub-data is a 1*H*W feature, corresponding to a feature of C=2, and so on, the cth segmented sub-data is a 1*H*W feature, corresponding to a feature of C=c.

[0090] Step 524, determine whether the product of the data volume of the first dimension and the data volume of all dimensions in the dimension set is greater than the upper limit data volume, if so, execute step 525, if not, execute step 526.

[0091] Assuming that the first dimension is C dimension, the dimension set includes H dimension and W dimension. In step 524, see Figure 5B As shown, it is necessary to determine whether C*H*W is greater than the upper limit data volume. The upper limit data volume is the upper limit data volume of the first internal storage medium, indicating that the first internal storage medium can store data of the upper limit data volume at most. For an introduction to the first internal storage medium, please refer to the subsequent embodiments.

[0092] Step 525: determine all dimensions in the dimension set as dimensions to be split.

[0093] For example, the H dimension and the W dimension may be determined as the dimensions to be segmented, that is, segmentation may be performed in the H dimension and the W dimension, that is, the input features may be segmented in the H dimension and the W dimension.

[0094] Exemplarily, the input feature is segmented in the H dimension and the W dimension, assuming that the input feature is a C*H*W feature, the input feature can be segmented into multiple segmented sub-data. For example, the first segmented sub-data is a C*h1*w1 feature, and the region corresponding to C channels is a h1*w1 feature, the second segmented sub-data is a C*h2*w2 feature, and the region corresponding to C channels is a h2*w2 feature, and so on. If H*W is divided into n regions, n segmented sub-data are obtained.

[0095] Step 526: The first dimension and all dimensions in the dimension set are determined as dimensions to be segmented. For example, the C dimension, the H dimension, and the W dimension can be determined as dimensions to be segmented, that is, the input features are segmented in the C dimension, the H dimension, and the W dimension.

[0096] Exemplarily, segmenting the input features in the C dimension, H dimension, and W dimension means that, assuming that the input feature is a C*H*W feature, the input feature can be segmented into multiple segmented sub-data. For example, the first segmented sub-data is a feature of 1*h1*w1, and the region corresponding to the first channel is a feature of h1*w1. The second segmented sub-data is a feature of 1*h1*w1, and the region corresponding to the second channel is a feature of h1*w1. By analogy, the cth segmented sub-data is a feature of 1*h1*w1, and the region corresponding to the cth channel is a feature of h1*w1. Then, the c+1th segmented sub-data is a feature of 1*h2*w2, and the region corresponding to the first channel is a feature of h2*w2, and so on. If H*W is divided into n regions, c*n segmented sub-data are obtained, that is, each channel corresponds to n segmented sub-data.

[0097] At this point, step 511 is completed, and a dimension to be segmented can be selected from multiple dimensions corresponding to the input feature, and the input feature is segmented on the dimension to be segmented to obtain multiple segmented sub-data.

[0098] Step 512: The input feature is stored in an external storage medium of the processor (ie, located outside the processor), and the first segmentation sub-data is traversed and stored in a first internal storage medium of the processor (located inside the processor).

[0099] Exemplarily, the internal storage medium of the processor may be divided into a first internal storage medium and a second internal storage medium, and the target unit operator may include a first computing unit and a second computing unit. The first computing unit may correspond to the first internal storage medium of the processor (i.e., the first internal storage medium is used to store data required by the first computing unit), and the second computing unit may correspond to the second internal storage medium of the processor (i.e., the second internal storage medium is used to store data required by the second computing unit).

[0100] See also Figure 5C As shown, it is a schematic diagram of an internal storage medium and an external storage medium. The input features are stored in the external storage medium of the processor. When the input features are copied from the external storage medium to the internal storage medium of the processor, the input features need to be segmented to obtain multiple segmented sub-data. Based on this, each segmented sub-data is traversed in turn, and for the currently traversed segmented sub-data (recorded as the first segmented sub-data), the first segmented sub-data is stored in the first internal storage medium of the processor.

[0101] For example, since the storage resources of the processor's internal storage medium are limited, and the storage resources occupied by the input features are relatively large, if the processor's internal storage medium cannot store the input features, the input features can be segmented to obtain multiple segmented sub-data. In this way, the segmented sub-data can be stored in the processor's internal storage medium so that the segmented sub-data can be stored.

[0102] Exemplarily, for the target unit operator, the first computing unit may be a matrix computing unit, and the second computing unit may be a non-matrix computing unit; or, the first computing unit may be a non-matrix computing unit, and the second computing unit may be a matrix computing unit. The matrix computing unit may include a unit for performing matrix operations; and the non-matrix computing unit may include a unit for performing non-matrix operations.

[0103] Step 513: After storing the first cutting sub-data in the first internal storage medium, read the first cutting sub-data from the first internal storage medium by the first computing unit, process the first cutting sub-data to obtain first result sub-data, and store the first result sub-data in the second internal storage medium.

[0104] Exemplarily, if the first computing unit is a matrix computing unit, the matrix computing unit can perform matrix computing processing based on the first slice sub-data to obtain first result sub-data, or, if the first computing unit is a non-matrix computing unit, the non-matrix computing unit can perform non-matrix computing processing based on the first slice sub-data to obtain first result sub-data, and there is no restriction on this processing process.

[0105] Step 514: After storing the first result sub-data in the second internal storage medium, read the first result sub-data from the second internal storage medium by a second computing unit, process the first result sub-data to obtain second result sub-data, and store the second result sub-data in an external storage medium.

[0106] Exemplarily, if the second computing unit is a non-matrix computing unit, the non-matrix computing unit can perform non-matrix computing processing based on the first result sub-data to obtain second result sub-data, or, if the second computing unit is a matrix computing unit, the matrix computing unit can perform matrix computing processing based on the first result sub-data to obtain second result sub-data, and there is no restriction on this processing process.

[0107] Exemplarily, the second result sub-data represents the final operation result of the target unit operator for the first slice sub-data, and the second result sub-data can be stored in an external storage medium. Based on this, the second result sub-data corresponding to all the slice sub-data are used to generate the output features of the target unit operator. For example, the second result sub-data corresponding to all the slice sub-data can be concatenated to obtain the output features.

[0108] Step 515: After reading the first slice sub-data from the first internal storage medium through the first computing unit (i.e., the first slice sub-data in the first internal storage medium has been successfully read, and the first internal storage medium can continue to store new slice sub-data), continue to traverse the second slice sub-data (such as the next slice sub-data of the first slice sub-data as the second slice sub-data) and store them in the first internal storage medium.

[0109] After the second cutting sub-data is stored in the first internal storage medium, the second cutting sub-data is read from the first internal storage medium by the first computing unit, the first result sub-data is obtained by processing based on the second cutting sub-data, and the first result sub-data is stored in the second internal storage medium.

[0110] Exemplarily, when the second cutting sub-data is stored in the first internal storage medium, the second cutting sub-data may be updated to the first cutting sub-data, so that step 513 may be re-executed, the first cutting sub-data may be read from the first internal storage medium by the first computing unit, the first result sub-data may be obtained by processing based on the first cutting sub-data, and the first result sub-data may be stored in the second internal storage medium. This process may be repeated until the last cutting sub-data is stored in the first internal storage medium.

[0111] It can be seen from the above content that after the first segmentation sub-data is read from the first internal storage medium by the first computing unit, the first result sub-data can be stored in the second internal storage medium, and the second computing unit performs processing based on the first result sub-data. Also, after the first segmentation sub-data is read from the first internal storage medium by the first computing unit, the second segmentation sub-data can be stored in the first internal storage medium, and the first computing unit performs processing based on the second segmentation sub-data.

[0112] To sum up, the process of processing based on the second cut sub-data by the first computing unit (first reading the second cut sub-data from the first internal storage medium, and then processing based on the second cut sub-data) and the process of processing based on the first result sub-data by the second computing unit (first reading the first result sub-data from the second internal storage medium, and then processing based on the first result sub-data) can be executed in parallel, that is, these two processes can be executed simultaneously, thereby improving the processing efficiency of the processor.

[0113] For example, see Figure 5D As shown, it is a schematic diagram of enabling multiple pipelines of a processor at the same time. Assuming that the processor is a multi-core processor, the multi-core processor can support parallel processing of multiple pipelines.

[0114] Based on pipeline 1, the sliced ​​sub-data 1 is stored in the first internal storage medium, and the first computing unit performs matrix calculation based on the sliced ​​sub-data 1; the first result sub-data 1 is stored in the second internal storage medium, and the second computing unit performs non-matrix calculation based on the first result sub-data 1.

[0115] Based on pipeline 2, the sliced ​​sub-data 2 is stored in the first internal storage medium (it is necessary to wait for the first computing unit to complete the processing of the sliced ​​sub-data 1), the first computing unit performs matrix calculation based on the sliced ​​sub-data 2, the first result sub-data 2 is stored in the second internal storage medium, and the second computing unit performs non-matrix calculation based on the first result sub-data 2. Obviously, the process of the first computing unit performing matrix calculation based on the sliced ​​sub-data 2 and the process of the second computing unit performing non-matrix calculation based on the first result sub-data 1 can be executed in parallel, so that the two operations are executed at the same time.

[0116] Based on pipeline 3, the sliced ​​sub-data 3 are stored in the first internal storage medium, the first computing unit performs matrix calculation based on the sliced ​​sub-data 3, the first result sub-data 3 are stored in the second internal storage medium, and the second computing unit performs non-matrix calculation based on the first result sub-data 3. Obviously, the process of the first computing unit performing matrix calculation based on the sliced ​​sub-data 3 and the process of the second computing unit performing non-matrix calculation based on the first result sub-data 2 can be executed in parallel.

[0117] In the embodiment of the present application, a data processing method is proposed, which can be applied to a processor, see Figure 5E FIG. 1 is a flow chart of the data processing method, which may include:

[0118] Step 531: If the input feature of the target unit operator corresponds to multiple dimensions, a dimension to be segmented is selected from the multiple dimensions, and the input feature is segmented on the dimension to be segmented to obtain multiple segmented sub-data. Step 531 is similar to step 511 and will not be repeated here.

[0119] Step 532: The input feature is stored in an external storage medium of the processor (ie, located outside the processor), and the first segmentation sub-data is traversed and stored in a first internal storage medium of the processor (located inside the processor).

[0120] Step 533: After storing the first cutting sub-data in the first internal storage medium, read the first cutting sub-data from the first internal storage medium by the first computing unit, process the first cutting sub-data to obtain first result sub-data, and store the first result sub-data in the second internal storage medium.

[0121] Step 534: After storing the first result sub-data in the second internal storage medium, read the first result sub-data from the second internal storage medium by a second computing unit, process the first result sub-data to obtain second result sub-data, and store the second result sub-data in a third internal storage medium.

[0122] Exemplarily, in addition to the first internal storage medium and the second internal storage medium, the internal storage medium of the processor can also be divided into a third internal storage medium, and the first computing unit corresponds to the third internal storage medium of the processor (that is, the third internal storage medium is used to store data required by the first computing unit). It should be noted that the first internal storage medium is used to store data from the external storage medium, and the third internal storage medium is used to store data from the second computing unit. Based on this, after the second result sub-data is obtained by the second computing unit, the second result sub-data is stored in the third internal storage medium.

[0123] Step 535: After storing the second result sub-data in the third internal storage medium, read the second result sub-data from the third internal storage medium by the first computing unit, obtain third result sub-data based on the second result sub-data, and store the third result sub-data in the external storage medium.

[0124] Exemplarily, the third result sub-data represents the final operation result of the target unit operator for the first slice sub-data, and the third result sub-data can be stored in an external storage medium. Based on this, the third result sub-data corresponding to all the slice sub-data are used to generate the output features of the target unit operator. For example, the third result sub-data corresponding to all the slice sub-data can be concatenated to obtain the output features.

[0125] Step 536, after reading the first slice sub-data from the first internal storage medium by the first computing unit, continue to traverse the second slice sub-data and store them in the first internal storage medium. After storing the second slice sub-data in the first internal storage medium, read the second slice sub-data from the first internal storage medium by the first computing unit, process based on the second slice sub-data to obtain the first result sub-data, and store the first result sub-data in the second internal storage medium. When storing the second slice sub-data in the first internal storage medium, update the second slice sub-data to the first slice sub-data, so that step 533 is re-executed, the first slice sub-data is read from the first internal storage medium by the first computing unit, process based on the first slice sub-data to obtain the first result sub-data, and store the first result sub-data in the second internal storage medium. And so on, until the last slice sub-data is stored in the first internal storage medium.

[0126] It can be seen from the above content that after the first segmentation sub-data is read from the first internal storage medium by the first computing unit, the first result sub-data can be stored in the second internal storage medium, and the second computing unit performs processing based on the first result sub-data. Also, the second result sub-data is stored in the third internal storage medium by the second computing unit, and the second result sub-data is read from the third internal storage medium by the first computing unit, and the processing is performed based on the second result sub-data. Also, after the first segmentation sub-data is read from the first internal storage medium by the first computing unit, the second segmentation sub-data can be stored in the first internal storage medium, and the first computing unit performs processing based on the second segmentation sub-data.

[0127] To sum up, the process of processing based on the second cut sub-data by the first computing unit and the process of processing based on the first result sub-data by the second computing unit can be executed in parallel, that is, the two processes can be executed simultaneously, thereby improving the processing efficiency of the processor.

[0128] In addition, the process of processing based on the second cut sub-data by the first computing unit (first reading the second cut sub-data from the first internal storage medium, and then processing based on the second cut sub-data) and the process of processing based on the second result sub-data by the first computing unit (first reading the second result sub-data from the third internal storage medium, and then processing based on the second result sub-data) can be executed in parallel, that is, the two processes can be executed simultaneously, thereby improving the processing efficiency of the processor.

[0129] For example, see Fig. 5F As shown, it is a schematic diagram of enabling multiple pipelines of a processor at the same time. Assuming that the processor is a multi-core processor, the multi-core processor can support parallel processing of multiple pipelines.

[0130] Based on pipeline 1, the sliced ​​sub-data 1 is stored in the first internal storage medium, the first computing unit performs matrix calculation based on the sliced ​​sub-data 1, the first result sub-data 1 is stored in the second internal storage medium, the second computing unit performs non-matrix calculation based on the first result sub-data 1, the second result sub-data 1 is stored in the third internal storage medium, and the first computing unit performs matrix calculation based on the second result sub-data 1.

[0131] Based on pipeline 2, the slice sub-data 2 is stored in the first internal storage medium (it is necessary to wait for the first computing unit to complete the processing of the slice sub-data 1), the first computing unit performs matrix calculation based on the slice sub-data 2, and stores the first result sub-data 2 in the second internal storage medium, the second computing unit performs non-matrix calculation based on the first result sub-data 2, and stores the second result sub-data 2 in the third internal storage medium, and the first computing unit performs matrix calculation based on the second result sub-data 2.

[0132] Obviously, the process in which the first computing unit performs matrix calculation based on the sliced ​​sub-data 2 and the process in which the second computing unit performs non-matrix calculation based on the first result sub-data 1 can be executed in parallel, so that the two operations are executed in the same time. In addition, since the duration of the non-matrix calculation is less than the duration of the matrix calculation, the process in which the first computing unit performs matrix calculation based on the sliced ​​sub-data 2 and the process in which the first computing unit performs matrix calculation based on the second result sub-data 1 can be executed in parallel.

[0133] In the above embodiment, a double cache strategy can be supported, which means that the first internal storage medium of the first computing unit and the second internal storage medium of the second computing unit can be double cached, that is, the data storage processes of the two do not affect each other. The first internal storage medium of the first computing unit and the third internal storage medium of the first computing unit can be double cached, that is, the data storage processes of the two do not affect each other.

[0134] In a possible implementation, the target unit operator is any operator that includes both a matrix calculation unit and a non-matrix calculation unit, and the type of the target unit operator is not limited. For example, the target unit operator may include but is not limited to a self-attention unit operator (Self Attention, referred to as SA). Of course, the self-attention unit operator is only an example, and the self-attention unit operator will be used as an example for explanation later.

[0135] Exemplarily, the self-attention unit operator may include but is not limited to a first MatMul (matrix multiplication) process, a Scale process, a Mask process, a Softmax process, and a second MatMul (matrix multiplication) process. For example, see Fig. 6A As shown, it is a structural diagram of the self-attention unit operator. The first MatMul process can be recorded as MatMul_1, and the second MatMul process can be recorded as MatMul_2. Of course, Fig. 6A This is just an example and is not intended to limit this structure.

[0136] The self-attention unit operator can be applied to the Transformer network model as well as other network models. Taking the Transformer network model as an example, the Transformer network model is the target network model, and the self-attention unit operator is the target unit operator in the Transformer network model.

[0137] The Transformer network model is a neural network architecture for sequence-to-sequence tasks, which is completely based on the self-attention mechanism without using recursive or convolutional structures. The Transformer network model is based on the Encoder and Decoder architecture, and uses the Self-Attention Unit operator (Self-Attention), Point-Wise (Point-Wise refers to the separate operation of each element in the data) and the fully connected layer in both the Encoder and the Decoder. There is no restriction on the structure of the Transformer network model.

[0138] In the Transformer network model, the self-attention unit operator can aggregate information by assigning different weights to the input information according to the current query, which is a kind of "weighted average" in terms of operation. The input features of the self-attention unit operator can include query vector, keyword vector and value vector. The query vector is recorded as Q (Query), the keyword vector is recorded as K (Key), and the value vector is recorded as V (Value).

[0139] For a given query vector, k keyword vectors are matched by inner product calculation, and the inner product is normalized by Softmax to obtain k weights. Then, the output of the attention of the query vector is the weighted average of the value vectors corresponding to the k keyword vectors. For a series of N query vectors, the attention output of the N query vectors can be calculated by matrix, and there is no restriction on this process.

[0140] For example, the following formula is used to determine the Attention output of the query vector: , is a scaling factor to avoid the variance effect caused by the dot product.

[0141] In the embodiment of the present application, a data processing method is proposed, which can be applied to a processor, see Figure 6B FIG. 1 is a flow chart of the data processing method, which may include:

[0142] Step 611: If the input features of the self-attention unit operator (i.e., query vector, keyword vector, and value vector) correspond to multiple dimensions, a dimension to be split is selected from the multiple dimensions, and the input features are split on the dimension to be split to obtain multiple split sub-data.

[0143] Exemplarily, the input features of the self-attention unit operator include a query vector, a keyword vector, and a value vector. The query vector may have a length of K and a height of M, the keyword vector may have a length of N and a height of K, and the value vector may have a length of K and a height of N. Based on this, the multiple dimensions corresponding to the input features may include K, M, and N dimensions. Among them, the K dimension indicates how many pixels there are in the length direction of the query vector, the M dimension indicates how many pixels there are in the height direction of the query vector, and the N dimension indicates how many pixels there are in the length direction of the keyword vector.

[0144] Exemplarily, the dimension with the largest amount of data can be used as the first dimension, and the remaining dimensions can be grouped into a dimension set. For example, the first dimension is the K dimension, and all dimensions in the dimension set are the M dimension and the N dimension; or, the first dimension is the M dimension, and all dimensions in the dimension set are the K dimension and the N dimension; or, the first dimension is the N dimension, and all dimensions in the dimension set are the K dimension and the M dimension.

[0145] Exemplarily, it can be determined whether the data volume of the first dimension is greater than the product of the data volumes of all dimensions in the dimension set. If so, the first dimension can be determined as the dimension to be split. If not, it is further determined whether the product of the data volume of the first dimension and the data volume of all dimensions in the dimension set is greater than the upper limit data volume. If so, all dimensions in the dimension set can be determined as the dimensions to be split. If not, the first dimension and all dimensions in the dimension set can be determined as the dimensions to be split.

[0146] For example, if the first dimension is K dimension, determine whether the data volume of K dimension is greater than the product of the data volume of M dimension and the data volume of N dimension. If so, determine the K dimension as the dimension to be split. If not, determine whether the product of the data volume of K dimension, the data volume of M dimension and the data volume of N dimension is greater than the upper limit data volume. If yes, the M dimension and N dimension can be determined as the dimensions to be split. If not, the K dimension, M dimension and N dimension can be determined as the dimensions to be split. The implementation process for the first dimension being M dimension or N dimension is similar to the implementation process for the first dimension being K dimension, and will not be repeated here.

[0147] Assuming that the dimension to be split includes K dimensions, the input features can be split in the K dimensions. Assuming that the dimension to be split includes M dimensions, the input features can be split in the M dimensions. Assuming that the dimension to be split includes N dimensions, the input features can be split in the N dimensions.

[0148] Step 612: The input feature is stored in an external storage medium of the processor (ie, located outside the processor), and the first segmentation sub-data is traversed and stored in a first internal storage medium of the processor (located inside the processor).

[0149] Exemplarily, the self-attention unit operator may include a matrix calculation unit and a non-matrix calculation unit, the matrix calculation unit may correspond to the first internal storage medium and the third internal storage medium of the processor, and the non-matrix calculation unit may correspond to the second internal storage medium of the processor. Based on this, each slice sub-data is traversed in turn, and the first slice sub-data currently traversed is stored in the first internal storage medium of the processor.

[0150] Step 613: After storing the first slice sub-data in the first internal storage medium, read the first slice sub-data from the first internal storage medium through the matrix calculation unit, perform first MatMul processing, Scale processing and Mask processing based on the first slice sub-data to obtain first result sub-data, and store the first result sub-data in the second internal storage medium through the matrix calculation unit.

[0151] Step 614: After storing the first result sub-data in the second internal storage medium, read the first result sub-data from the second internal storage medium through the non-matrix calculation unit, perform Softmax processing based on the first result sub-data to obtain second result sub-data, and store the second result sub-data in a third internal storage medium.

[0152] Step 615: After storing the second result subdata in the third internal storage medium, read the second result subdata from the third internal storage medium through the matrix calculation unit, perform second MatMul processing based on the second result subdata to obtain third result subdata, and store the third result subdata in the external storage medium.

[0153] Step 616: After the first slice sub-data is read from the first internal storage medium by the matrix calculation unit, the second slice sub-data is traversed and stored in the first internal storage medium. After the second slice sub-data is stored in the first internal storage medium, the second slice sub-data is read from the first internal storage medium by the matrix calculation unit, and the first result sub-data is obtained by processing based on the second slice sub-data, and the first result sub-data is stored in the second internal storage medium, and so on, and the above steps are repeated.

[0154] For example, see Figure 6C As shown in the figure, it is a processing diagram of the self-attention unit operator. The K dimension of the attention unit operator is small, so the data volume of the two matrices, the keyword vector and the value vector, is small, and can be copied to the internal storage medium of the processor at one time. The M dimension of the attention unit operator is large, so the data volume of the query vector is large, and the M dimension of the query vector can be tiling (tiling) split, that is, the M dimension of the query vector is used as the dimension to be split, and multiple split sub-data should be obtained on the dimension to be split.

[0155] See also Figure 6CAs shown, the query vector Q can be segmented to obtain K*m first segmentation sub-data, that is, the M dimension of the query vector is segmented to obtain sub-data of height m. Then, the K*m first segmentation sub-data and the keyword vector (N*K data) are stored in the first internal storage medium.

[0156] The matrix calculation unit performs MatMul_1 (first matrix multiplication) processing, Scale processing and Mask processing based on the K*m first slice sub-data and the keyword vector K to obtain the N*m first result sub-data MM1. Fig. 6A As shown in the figure, MatMul_1 processing is used to perform matrix multiplication based on the query vector Q (the first slice of K*m) and the keyword vector K, thereby realizing the fully connected layer. There is no restriction on this processing process. Scale processing is used to perform multiplication operations, multiplying the result of MatMul_1 by a constant. Mask processing is an addition operation, adding a mask value to the result of MatMul_1 element by element.

[0157] The N*m first result sub-data MM1 is stored in the second internal storage medium. The non-matrix calculation unit performs Softmax processing based on the N*m first result sub-data MM1 to obtain the N*m second result sub-data MM2. The Softmax processing is performed on the N dimension. Since the N dimension is not segmented when the data is segmented, the Softmax processing result can be easily obtained.

[0158] The N*m second result sub-data MM2 and the value vector (K*N data) are stored in the third internal storage medium. The matrix calculation unit performs MatMul_2 (second matrix multiplication) processing based on the N*m second result sub-data MM2 and the value vector V to obtain the K*m third result sub-data Y. Fig. 6A As shown, the MatMul_2 process is used to perform a matrix multiplication operation based on the N*m second result sub-data MM2 and the value vector V.

[0159] The third result sub-data Y of K*m can be stored in an external storage medium (after the third result sub-data Y is stored in the external storage medium, post-processing such as quantization and biasing can be added). The third result sub-data Y represents the final operation result of the self-attention unit operator for the first slice sub-data, and the third result sub-data Y corresponding to all the slice sub-data can constitute the output features of K*N.

[0160] Furthermore, the query vector Q is further segmented to obtain K*m first segmentation sub-data, and the above steps are repeated to obtain a new third result sub-data Y which is stored in an external storage medium, and so on.

[0161] Exemplarily, after the query vector Q is split, the matrix computing units and non-matrix computing units can be executed in parallel in a pipeline, that is, multiple loops are performed on the key subgraph structure, a small portion of data is processed each time, and the internal storage medium is accessed in an approximately streaming manner, thereby reducing the total amount of data accessed by the memory, and better hiding the overlap of computing and data handling, thereby achieving performance optimization of time-consuming acceleration.

[0162] For example, the application of pipeline parallel strategy in the self-attention unit operator can be: In the execution of the self-attention unit operator, it can mainly include the following three stages: Stage 1: MatMul_1, Stage 2: Softmax, Stage 3: MatMul_2. MatMul_1 and MatMul_2 involve a lot of memory reading and writing, which are IO-intensive operations, while the data of Softmax operation is stored in the internal storage medium, which is a computation-intensive operation. In summary, it can be seen that the memory access time of the MatMul stage needs to be hidden as much as possible.

[0163] Based on this, Softmax can be split into two parts, which are executed concurrently with MatMul, see Fig.6D The figure shows a schematic diagram of the concurrent execution of Softmax and MatMul. For example, the execution of the self-attention unit operator can be re-divided into two stages: Stage 1: MatMul_1 (first matrix multiplication processing) + Softmax_part1 (part 1 of soft maximum processing), Stage 2: Softmax_part2 (part 2 of soft maximum processing) + MatMul_2 (second matrix multiplication processing). In other words, the calculation of Softmax is divided into two parts, which are assigned to MatMul_1 and MatMul_2 respectively, and the calculation and handling are accelerated in parallel in the two stages, and different computing units are used efficiently, so as to fully hide the IO time.

[0164] In summary, see Fig. 5FAs shown, based on pipeline 1, the sliced ​​sub-data 1 is stored in the first internal storage medium, the matrix calculation unit performs the MatMul_1 operation based on the sliced ​​sub-data 1, and the first result sub-data 1 is stored in the second internal storage medium, the non-matrix calculation unit performs the Softmax operation based on the first result sub-data 1, and the second result sub-data 1 is stored in the third internal storage medium, and the matrix calculation unit performs the MatMul_2 operation based on the second result sub-data 1. Based on pipeline 2, the sliced ​​sub-data 2 is stored in the first internal storage medium, the matrix calculation unit performs the MatMul_1 operation based on the sliced ​​sub-data 2, and the first result sub-data 2 is stored in the second internal storage medium, the non-matrix calculation unit performs the Softmax operation based on the first result sub-data 2, and the second result sub-data 2 is stored in the third internal storage medium, and the matrix calculation unit performs the MatMul_2 operation based on the second result sub-data 2.

[0165] Obviously, the process of the matrix calculation unit performing the MatMul_1 operation based on the sliced ​​sub-data 2 and the process of the non-matrix calculation unit performing the Softmax operation based on the first result sub-data 1 can be executed in parallel.

[0166] The process in which the matrix calculation unit performs the MatMul_1 operation based on the sliced ​​sub-data 2 and the process in which the matrix calculation unit performs the MatMul_2 operation based on the second result sub-data 1 may be executed in parallel.

[0167] Exemplarily, after the query vector Q is segmented, based on the reasoning time-consuming optimization solution (ie, the time-consuming acceleration optimization strategy), a bandwidth optimization strategy can also be implemented to optimize the matrix calculation segmentation strategy.

[0168] If the matrix calculation specification is [M, K, N], that is, the three dimensions corresponding to the input features (K dimension, M dimension and N dimension), the segmentation value of the segmented sub-data obtained each time is [m, k, n], then the amount of data accessed for each calculation is m*k+k*n, and a total of M / m*K / k*N / n cycles are required. The amount of data that needs to be accessed for one matrix calculation is: .

[0169] From the above formula, we can find that the memory access volume of the matrix calculation unit has nothing to do with the tiling size (i.e., the segmentation size) in the K dimension, but only with the tiling size in the M and N dimensions. The smaller the sum of the reciprocals of the two, the smaller the memory access volume, and theoretically the lower the bandwidth occupancy. Since the memory space of the processor's internal storage medium is limited, m and n obviously cannot be infinite. Based on this, since the amount of data calculated each time is m*k+k*n, and the amount of data is limited by the memory space of the internal storage medium, it can be preliminarily inferred that if you want to minimize the bandwidth occupancy, you can reduce the value of k, increase the values ​​of m and n, and set m≈n.

[0170] In summary, in the process of segmenting the input features on the dimension to be segmented to obtain multiple segmented sub-data, in order to implement the bandwidth optimization strategy, the following methods can be used:

[0171] If the dimensions to be split include both M and N dimensions, and the dimensions to be split include K dimensions, the value of k can be reduced, and the values ​​of m and n can be increased, and the absolute value of the difference between the split value m of the M dimension and the split value n of the N dimension is less than the first threshold (configured based on experience), such as m≈n.

[0172] If the dimension to be segmented includes both M and N dimensions, and the dimension to be segmented does not include K dimension, the value of k is equal to the specification value K of K dimension. The values ​​of m and n can be increased, and the absolute value of the difference between the segmentation value m of M dimension and the segmentation value n of N dimension is less than the first threshold, such as m≈n.

[0173] If the dimension to be segmented includes the M dimension but not the N dimension, and the dimension to be segmented may include the K dimension, the value of k may be reduced. Since the dimension to be segmented does not include the N dimension, the value of n is equal to the specification value N of the N dimension. Based on this, the absolute value of the difference between the segmentation value m of the M dimension and the specification value N of the N dimension is less than the second threshold value (configured according to experience), such as m≈N.

[0174] If the dimension to be segmented includes the M dimension but does not include the N dimension, and the dimension to be segmented does not include the K dimension, the value of k is equal to the specification value K of the K dimension. Since the dimension to be segmented does not include the N dimension, the value of n is equal to the specification value N of the N dimension. Based on this, the absolute value of the difference between the segmentation value m of the M dimension and the specification value N of the N dimension is less than the second threshold (configured according to experience), such as m≈N.

[0175] If the dimension to be segmented includes N dimension but not M dimension, and the dimension to be segmented may include K dimension, the value of k may be reduced. Since the dimension to be segmented does not include M dimension, the value of m is equal to the specification value M of M dimension. Based on this, the absolute value of the difference between the segmentation value n of N dimension and the specification value M of M dimension is less than the third threshold value (configured according to experience), such as n≈M.

[0176] If the dimension to be segmented includes the N dimension but does not include the M dimension, and the dimension to be segmented does not include the K dimension, the value of k is equal to the specification value K of the K dimension. Since the dimension to be segmented does not include the M dimension, the value of m is equal to the specification value M of the M dimension. Based on this, the absolute value of the difference between the segmentation value n of the N dimension and the specification value M of the M dimension is less than the third threshold value (configured according to experience), such as n≈M.

[0177] If the dimension to be segmented does not include the N dimension and does not include the M dimension, but includes the K dimension, the value of k can be reduced. In addition, since the dimension to be segmented does not include the M dimension, the value of m is equal to the specification value M of the M dimension. Since the dimension to be segmented does not include the N dimension, the value of n is equal to the specification value N of the N dimension.

[0178] As can be seen from the above technical solutions, in the embodiment of the present application, a Transformer network deployment method for image data and video data is proposed, which mainly includes unified graph representation, graph optimization and graph compilation processes. A method for implementing pipeline parallel acceleration for matrix computing units and non-matrix computing units is proposed. Combined with the graph optimization method, the data to be calculated in the Transformer network is accelerated in parallel on the matrix computing units and non-matrix computing units, while ensuring execution efficiency and taking bandwidth optimization into account. For the core subgraph Self-Attention of Transformer, an operator graph acceleration strategy is proposed to optimize the transformer network reasoning time and bandwidth occupancy rate, and the segmentation strategy of the tiling basic block is used to calculate and transport pipeline parallelism, etc., to optimize the reasoning time and bandwidth occupancy rate, which is not only applicable to the time optimization and bandwidth optimization under the unfused single operator, but also applicable to the time optimization and bandwidth optimization under the fusion of multiple operators. For hardware platforms with limited internal storage medium resources (such as end-side platforms), a universal Transformer network deployment method is proposed, but the method is still highly portable and scalable for non-end-side platforms.

[0179] Based on the same application concept as the above method, a data processing device is proposed in an embodiment of the present application, which is applied to a processor, wherein the processor deploys a target network model, wherein the target network model includes a target unit operator, wherein the target unit operator includes a first computing unit and a second computing unit, wherein the first computing unit corresponds to a first internal storage medium located in the processor, and the second computing unit corresponds to a second internal storage medium located in the processor, see Figure 7 FIG. 1 is a schematic diagram of the structure of the device, and the device may include:

[0180] A selection module 71, configured to select a dimension to be segmented from the multiple dimensions if the input feature of the target unit operator corresponds to multiple dimensions, and the input feature is stored in an external storage medium;

[0181] The processing module 72 is used to segment the input feature in the dimension to be segmented to obtain a plurality of segmented sub-data; traverse the first segmented sub-data and store them in a first internal storage medium;

[0182] The control module 73 is used to read the first cutting sub-data from the first internal storage medium through the first calculation unit, process the first cutting sub-data to obtain the first result sub-data, and store the first result sub-data in the second internal storage medium; read the first result sub-data from the second internal storage medium through the second calculation unit, and process the first result sub-data to obtain the second result sub-data;

[0183] The processing module 72 is further configured to traverse the second segmentation sub-data and store them in the first internal storage medium after the control module reads the first segmentation sub-data from the first internal storage medium through the first calculation unit;

[0184] The control module 73 is further configured to read the second cutting sub-data from the first internal storage medium through the first calculation unit, and obtain the first result sub-data by processing based on the second cutting sub-data;

[0185] The process of processing based on the second cutting sub-data by the first computing unit and the process of processing based on the first result sub-data by the second computing unit are executed in parallel.

[0186] Exemplarily, when the selection module 71 selects the dimension to be split from multiple dimensions, it is specifically used to: select the first dimension with the largest data volume from multiple dimensions; if the data volume of the first dimension is greater than the product of the data volumes of all dimensions in the dimension set, the first dimension is determined as the dimension to be split; wherein the dimension set includes at least two dimensions, and the dimension set includes all dimensions or part of dimensions except the first dimension; if the data volume of the first dimension is not greater than the product of the data volumes of all dimensions in the dimension set, and the product of the data volume of the first dimension and the data volume of all dimensions in the dimension set is greater than the upper limit data volume of the first internal storage medium, all dimensions in the dimension set are determined as the dimensions to be split; if the data volume of the first dimension is not greater than the product of the data volumes of all dimensions in the dimension set, and the product of the data volume of the first dimension and the data volume of all dimensions in the dimension set is not greater than the upper limit data volume, the first dimension and all dimensions in the dimension set are determined as the dimensions to be split.

[0187] Exemplarily, the processing module 72 is further used to store the second result sub-data in the external storage medium after obtaining the second result sub-data; wherein the second result sub-data corresponding to all the cut sub-data are used to generate the output characteristics of the target unit operator; or, the control module 73 is further used to store the second result sub-data in the third internal storage medium through the second calculation unit after obtaining the second result sub-data, wherein the first calculation unit corresponds to the third internal storage medium located in the processor; the second result sub-data is read by the first calculation unit, and the third result sub-data is obtained by processing based on the second result sub-data; wherein the process of processing based on the second cut sub-data by the first calculation unit is executed in parallel with the process of processing based on the second result sub-data by the first calculation unit; the processing module 72 is further used to store the third result sub-data in the external storage medium after obtaining the third result sub-data; wherein the third result sub-data corresponding to all the cut sub-data are used to generate the output characteristics of the target unit operator.

[0188] Exemplarily, the first computing unit includes a matrix computing unit, and the second computing unit includes a non-matrix computing unit; or, the first computing unit includes a non-matrix computing unit, and the second computing unit includes a matrix computing unit; wherein the matrix computing unit includes a unit for performing matrix operations; and the non-matrix computing unit includes a unit for performing non-matrix operations.

[0189] Exemplarily, if the target unit operator includes a self-attention unit operator, the self-attention unit operator includes a first MatMul processing, a Scale processing, a Mask processing, a Softmax processing and a second MatMul processing; the control module 73 is also used to read the first cut sub-data from the first internal storage medium through the first computing unit, perform a first MatMul processing, a Scale processing and a Mask processing based on the first cut sub-data to obtain first result sub-data, and store the first result sub-data in the second internal storage medium; read the first result sub-data from the second internal storage medium through the second computing unit, perform a Softmax processing based on the first result sub-data to obtain second result sub-data, and store the second result sub-data in the third internal storage medium; read the second result sub-data from the third internal storage medium through the first computing unit, perform a second MatMul processing based on the second result sub-data to obtain third result sub-data.

[0190] Exemplarily, the input features of the self-attention unit operator include a query vector, a keyword vector and a value vector, the query vector has a length of K and a height of M, the keyword vector has a length of N and a height of K, and the value vector has a length of K and a height of N; the multiple dimensions corresponding to the input features include K dimensions, M dimensions and N dimensions; wherein the first dimension is K dimension, and all dimensions in the dimension set are M dimensions and N dimensions; or, the first dimension is M dimension, and all dimensions in the dimension set are K dimensions and N dimensions; or, the first dimension is N dimension, and all dimensions in the dimension set are K dimensions and N dimensions.

[0191] Exemplarily, in the process of segmenting the input feature on the dimension to be segmented to obtain multiple segmentation sub-data, if the dimension to be segmented includes both M dimension and N dimension, the absolute value of the difference between the segmentation value m of the M dimension and the segmentation value n of the N dimension is less than a first threshold; or, if the dimension to be segmented includes M dimension but does not include N dimension, the absolute value of the difference between the segmentation value m of the M dimension and the specification value of the N dimension is less than a second threshold; or, if the dimension to be segmented includes N dimension but does not include M dimension, the absolute value of the difference between the segmentation value n of the N dimension and the specification value of the M dimension is less than a third threshold.

[0192] Based on the same application concept as the above method, an electronic device is proposed in the embodiment of the present application, see Figure 8 As shown, the electronic device includes: a processor 81 and a machine-readable storage medium 82, wherein the machine-readable storage medium 82 stores machine-executable instructions that can be executed by the processor 81; the processor 81 is used to execute the machine-executable instructions to implement the data processing method disclosed in the above example of this application.

[0193] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the data processing method disclosed in the above example of the present application can be implemented.

[0194] Based on the same application concept as the above method, the embodiment of the present application further provides a computer program product, which may include a computer program. When the computer program is executed by a processor, the data processing method disclosed in the above example of the present application is implemented.

[0195] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A data processing method, characterized in that: Applied to a processor, the processor deploys a target network model, the target network model includes a target unit operator, the target unit operator includes a first computing unit and a second computing unit, the first computing unit corresponds to a first internal storage medium of the processor, and the second computing unit corresponds to a second internal storage medium of the processor, the method includes: If the input feature of the target unit operator corresponds to multiple dimensions, a dimension to be segmented is selected from the multiple dimensions, and the input feature is segmented on the dimension to be segmented to obtain multiple segmented sub-data; wherein the input feature is stored in an external storage medium; Traversing the first cutting sub-data and storing them in a first internal storage medium; reading the first cutting sub-data by a first computing unit, processing the first cutting sub-data to obtain first result sub-data, and storing the first result sub-data in a second internal storage medium; reading the first result sub-data by a second computing unit, processing the first result sub-data to obtain second result sub-data; After reading the first segmentation sub-data by the first computing unit, traversing the second segmentation sub-data and storing them in the first internal storage medium, returning to execute the step of reading the second segmentation sub-data by the first computing unit, and processing the second segmentation sub-data to obtain the first result sub-data; The process of processing based on the second cutting sub-data by the first computing unit and the process of processing based on the first result sub-data by the second computing unit are executed in parallel; The step of selecting a dimension to be segmented from the multiple dimensions includes: Selecting a first dimension with the largest amount of data from the multiple dimensions; If the data volume of the first dimension is greater than the product of the data volumes of all dimensions in the dimension set, the first dimension is determined as the dimension to be segmented; wherein the dimension set includes at least two dimensions, and the dimension set includes all dimensions or part of dimensions except the first dimension; If the data volume of the first dimension is not greater than the product of the data volumes of all dimensions in the dimension set, and the product of the data volume of the first dimension and the data volume of all dimensions in the dimension set is greater than the upper limit data volume of the first internal storage medium, all dimensions in the dimension set are determined as the dimensions to be segmented; If the data volume of the first dimension is not greater than the product of the data volume of all dimensions in the dimension set, and the product of the data volume of the first dimension and the data volume of all dimensions in the dimension set is not greater than the upper limit data volume, then the first dimension and all dimensions in the dimension set are determined as the dimensions to be split.

2. The method according to claim 1, characterized in that After the first result sub-data is read by the second calculation unit and the second result sub-data is obtained by processing based on the first result sub-data, the method further includes: storing the second result sub-data in the external storage medium; wherein the second result sub-data corresponding to all the slice sub-data are used to generate the output features of the target unit operator; or, The second result sub-data is stored in a third internal storage medium by a second computing unit, wherein the first computing unit corresponds to the third internal storage medium located in the processor; the second result sub-data is read by the first computing unit, and the third result sub-data is obtained by processing based on the second result sub-data; wherein the process of processing based on the second cutting sub-data by the first computing unit and the process of processing based on the second result sub-data by the first computing unit are executed in parallel; The third result sub-data is stored in the external storage medium; the third result sub-data corresponding to all the slice sub-data are used to generate the output features of the target unit operator.

3. The method according to claim 1 or 2, characterized in that: The first computing unit includes a matrix computing unit, and the second computing unit includes a non-matrix computing unit; or, the first computing unit includes a non-matrix computing unit, and the second computing unit includes a matrix computing unit; Wherein, the matrix calculation unit includes a unit for performing matrix operations; The non-matrix computing unit includes a unit for performing non-matrix operations.

4. The method according to claim 1 or 2, characterized in that: If the target unit operator includes a self-attention unit operator, and the self-attention unit operator includes a first MatMul process, a Scale process, a Mask process, a Softmax process, and a second MatMul process, the method further includes: Reading first segmentation sub-data from a first internal storage medium through a first computing unit, performing first MatMul processing, Scale processing, and Mask processing on the first segmentation sub-data to obtain first result sub-data, and storing the first result sub-data in a second internal storage medium; Reading the first result sub-data from the second internal storage medium by a second calculation unit, performing Softmax processing based on the first result sub-data to obtain second result sub-data, and storing the second result sub-data in a third internal storage medium; The second result sub-data is read from the third internal storage medium by the first calculation unit, and a second MatMul process is performed based on the second result sub-data to obtain third result sub-data.

5. The method according to claim 4, characterized in that The input features of the self-attention unit operator include a query vector, a keyword vector and a value vector, the query vector has a length of K and a height of M, the keyword vector has a length of N and a height of K, and the value vector has a length of K and a height of N; the multiple dimensions corresponding to the input features include K dimension, M dimension and N dimension; The first dimension is K dimension, and all dimensions in the dimension set are M dimension and N dimension; or, the first dimension is M dimension, and all dimensions in the dimension set are K dimension and N dimension; or, the first dimension is N dimension, and all dimensions in the dimension set are K dimension and M dimension.

6. The method according to claim 5, characterized in that In the process of segmenting the input feature on the to-be-segmented dimension to obtain a plurality of segmented sub-data, If the to-be-divided dimensions include both M and N dimensions, the absolute value of the difference between the segmentation value m of the M dimension and the segmentation value n of the N dimension is less than a first threshold; Alternatively, if the to-be-divided dimensions include the M dimension but not the N dimension, the absolute value of the difference between the segmentation value m of the M dimension and the specification value of the N dimension is less than the second threshold; Alternatively, if the to-be-divided dimensions include N dimensions but do not include M dimensions, the absolute value of the difference between the segmentation value n of the N dimensions and the specification value of the M dimensions is less than a third threshold.

7. A data processing device, characterized in that: Applied to a processor, the processor deploys a target network model, the target network model includes a target unit operator, the target unit operator includes a first computing unit and a second computing unit, the first computing unit corresponds to a first internal storage medium located in the processor, and the second computing unit corresponds to a second internal storage medium located in the processor, the device includes: A selection module, configured to select a dimension to be segmented from the multiple dimensions if the input feature of the target unit operator corresponds to multiple dimensions, and the input feature is stored in an external storage medium; A processing module is used to segment the input feature on the to-be-segmented dimension to obtain a plurality of segmented sub-data; traverse the first segmented sub-data and store them in a first internal storage medium; A control module, configured to read first segmentation sub-data from a first internal storage medium through a first computing unit, obtain first result sub-data based on the first segmentation sub-data by processing, and store the first result sub-data in a second internal storage medium; read the first result sub-data from a second internal storage medium through a second computing unit, obtain second result sub-data based on the first result sub-data by processing; The processing module is further configured to traverse the second segmentation sub-data and store them in the first internal storage medium after the control module reads the first segmentation sub-data from the first internal storage medium through the first calculation unit; The control module is further configured to read the second cutting sub-data from the first internal storage medium through the first computing unit, and obtain the first result sub-data by processing based on the second cutting sub-data; The process of processing based on the second cutting sub-data by the first computing unit and the process of processing based on the first result sub-data by the second computing unit are executed in parallel; Among them, when the selection module selects the dimension to be split from the multiple dimensions, it is specifically used to: select the first dimension with the largest data volume from the multiple dimensions; if the data volume of the first dimension is greater than the product of the data volumes of all dimensions in the dimension set, the first dimension is determined as the dimension to be split; wherein the dimension set includes at least two dimensions, and the dimension set includes all dimensions or part of the dimensions except the first dimension; if the data volume of the first dimension is not greater than the product of the data volumes of all dimensions in the dimension set, and the product of the data volume of the first dimension and the data volume of all dimensions in the dimension set is greater than the upper limit data volume of the first internal storage medium, all dimensions in the dimension set are determined as the dimensions to be split; if the data volume of the first dimension is not greater than the product of the data volumes of all dimensions in the dimension set, and the product of the data volume of the first dimension and the data volume of all dimensions in the dimension set is not greater than the upper limit data volume, the first dimension and all dimensions in the dimension set are determined as the dimensions to be split.

8. The device according to claim 7, It is characterized in that in, The processing module is further used to store the second result sub-data to the external storage medium after obtaining the second result sub-data; wherein the second result sub-data corresponding to all the cut sub-data are used to generate the output characteristics of the target unit operator; or, the control module is further used to store the second result sub-data to a third internal storage medium through a second calculation unit after obtaining the second result sub-data, wherein the first calculation unit corresponds to the third internal storage medium located in the processor; the second result sub-data is read by the first calculation unit, and the third result sub-data is obtained by processing based on the second result sub-data; wherein the process of processing based on the second cut sub-data by the first calculation unit and the process of processing based on the second result sub-data by the first calculation unit are executed in parallel; the processing module is further used to store the third result sub-data to the external storage medium after obtaining the third result sub-data; wherein the third result sub-data corresponding to all the cut sub-data are used to generate the output characteristics of the target unit operator; Wherein, the first computing unit includes a matrix computing unit, and the second computing unit includes a non-matrix computing unit; or, the first computing unit includes a non-matrix computing unit, and the second computing unit includes a matrix computing unit; wherein the matrix computing unit includes a unit for performing matrix operations; and the non-matrix computing unit includes a unit for performing non-matrix operations; Wherein, if the target unit operator includes a self-attention unit operator, the self-attention unit operator includes a first MatMul processing, a Scale processing, a Mask processing, a Softmax processing, and a second MatMul processing; the control module is also used to read the first slice sub-data from the first internal storage medium through the first calculation unit, perform a first MatMul processing, a Scale processing, and a Mask processing based on the first slice sub-data to obtain first result sub-data, and store the first result sub-data in the second internal storage medium; read the first result sub-data from the second internal storage medium through the second calculation unit, perform a Softmax processing based on the first result sub-data to obtain second result sub-data, and store the second result sub-data in the third internal storage medium; read the second result sub-data from the third internal storage medium through the first calculation unit, perform a second MatMul processing based on the second result sub-data to obtain third result sub-data; The input features of the self-attention unit operator include a query vector, a keyword vector and a value vector, the query vector has a length of K and a height of M, the keyword vector has a length of N and a height of K, and the value vector has a length of K and a height of N; the multiple dimensions corresponding to the input features include K dimensions, M dimensions and N dimensions; the first dimension is K dimension, and all dimensions in the dimension set are M dimensions and N dimensions; or, the first dimension is M dimension, and all dimensions in the dimension set are K dimensions and N dimensions; or, the first dimension is N dimension, and all dimensions in the dimension set are K dimensions and M dimensions; Among them, in the process of segmenting the input feature on the dimension to be segmented to obtain multiple segmentation sub-data, if the dimension to be segmented includes both M dimension and N dimension, the absolute value of the difference between the segmentation value m of the M dimension and the segmentation value n of the N dimension is less than a first threshold; or, if the dimension to be segmented includes M dimension but does not include N dimension, the absolute value of the difference between the segmentation value m of the M dimension and the specification value of the N dimension is less than a second threshold; or, if the dimension to be segmented includes N dimension but does not include M dimension, the absolute value of the difference between the segmentation value n of the N dimension and the specification value of the M dimension is less than a third threshold.

9. An electronic device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method described in any one of claims 1-6.

Images