Data processing method and device and data processing chip

Abstract

The invention discloses a data processing method and device and a data processing chip, and relates to the technical field of artificial intelligence, the data processing chip comprises an operation component and a controller, the operation component comprises a processing unit array, the controller is used for obtaining a first data object and a second data object, the first data object comprises a plurality of first sub-objects, and the second data object comprises a plurality of second sub-objects; the first data object comprises a plurality of first sub-objects, the second data object comprises a plurality of second sub-objects, each first sub-object and each second sub-object correspond to a corresponding sub-object pair to be processed, and each first data contained in the first sub-object in the sub-object pair to be processed and the corresponding second data contained in the second sub-object correspond to a data pair to be processed; and in response to the first data or the second data in the to-be-processed data pair being invalid data, issuing indication information representing that the first data or the second data is invalid data to a processing unit corresponding to the to-be-processed sub-object pair, so that the processing unit skips data processing on the to-be-processed data pair based on the indication information.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, device, and data processing chip. Background Art

[0002] In current digital systems like SOCs (System on Chips), artificial intelligence (AI) has become a core function. System power consumption is primarily driven by processors like the Neural Network Processing Unit (NPU) that handle AI tasks. 80% of the power consumption of NPUs and other processors is consumed by computing components like the MAC array. Optimizing the power consumption of computing components is crucial for optimizing the power consumption of NPUs and other processors, and ultimately, the entire digital system. Summary of the Invention

[0003] To this end, this application discloses the following technical solutions:

[0004] A data processing chip, comprising:

[0005] A computing component, including an array of processing units;

[0006] Controller for:

[0007] Obtain a first data object and a second data object;

[0008] The first data object includes a plurality of first sub-objects, and the second data object includes a plurality of second sub-objects. Each first sub-object and each second sub-object correspond to a corresponding sub-object pair to be processed. Each first data contained in the first sub-object and the corresponding second data contained in the second sub-object in the sub-object pair to be processed correspond to a data pair to be processed. Each sub-object pair to be processed is used to be assigned to a corresponding processing unit in the computing component for data processing.

[0009] For each data pair to be processed in the sub-object pair to be processed, in response to the first data or the second data in the data pair to be processed being invalid data, indication information representing that the first data or the second data is invalid data is sent to the processing unit corresponding to the sub-object pair to be processed, so that the processing unit skips data processing of the data pair to be processed based on the indication information.

[0010] Optionally, the processing unit corresponding to each to-be-processed sub-object pair is configured to perform data processing on each to-be-processed data pair contained in the to-be-processed sub-object pair in a serial manner through a corresponding processing cycle; the processing unit processes one to-be-processed data pair in one processing cycle;

[0011] The data processing chip further includes a first main register unit group, a first backup register unit group, a second main register unit group, and a second backup register unit group; each processing unit group in the first direction of the processing unit array corresponds to a first main register unit and a first backup register unit, for processing a first sub-object in the first data object; each processing unit group in the second direction of the processing unit array corresponds to a second main register unit and a second backup register unit, for processing a second sub-object in the second data object;

[0012] The first main register unit can be used to store the first data to be processed by the corresponding first sub-object in the current processing cycle, and the first backup register unit can be used to store the first data corresponding to the corresponding first sub-object in the corresponding historical processing cycle; the second main register unit can be used to store the second data to be processed by the corresponding second sub-object in the current processing cycle, and the second backup register unit can be used to store the second data corresponding to the corresponding second sub-object in the corresponding historical processing cycle.

[0013] Optionally, the controller is further configured to:

[0014] In response to the switching processing cycle, data update processing is performed on the corresponding storage unit corresponding to the first sub-object, and data update processing is performed on the corresponding storage unit corresponding to the second sub-object, and based on the data in each storage unit after the update, the first data and the second data are distributed to the corresponding processing unit to form a pair of data to be processed in the corresponding processing unit and perform data processing on the pair of data to be processed.

[0015] Optionally, when performing data update processing on the corresponding register unit corresponding to the first sub-object, performing data update processing on the corresponding register unit corresponding to the second sub-object, and distributing the first data and the second data to the corresponding processing unit based on the updated data in each register unit, the controller is configured to:

[0016] In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being invalid data, not updating data in the primary register unit and the backup register unit corresponding to the corresponding sub-object, not allocating the current data to be processed to the corresponding processing unit in the current processing cycle, generating indication information indicating that the current data to be processed is invalid data, and issuing the indication information to the corresponding processing unit;

[0017] In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being valid data, updating the primary register unit corresponding to the corresponding sub-object based on at least the current data to be processed, and distributing the data in the primary register unit and the backup register unit corresponding to the corresponding sub-object to the corresponding processing unit, so that the processing unit selects required data from the data in the primary register unit and the backup register unit corresponding to the corresponding sub-object;

[0018] The corresponding sub-object is the corresponding first sub-object or the second sub-object.

[0019] Optionally, the processing unit includes:

[0020] a first receiving unit configured to receive first data allocated by the controller based on the corresponding first active register unit and first standby register unit, and second indication information indicating that second data to be processed in the current processing cycle is invalid data; and in response to not receiving the first data, not updating the data on the first data branch; in response to receiving the first data and the second indication information, updating the data on the first data branch based on the first data in the first standby register unit; and in response to receiving the first data and not receiving the second indication information, updating the data in the first data branch based on the first data in the first active register unit;

[0021] a second receiving unit configured to receive second data allocated by the controller based on the corresponding second active register unit and second standby register unit, and first indication information indicating that first data to be processed in the current processing cycle is invalid data; and in response to not receiving the second data, not updating the data on the second data branch; in response to receiving the second data and the first indication information, updating the data on the second data branch based on the second data in the second standby register unit; and in response to receiving the second data and not receiving the first indication information, updating the data in the second data branch based on the second data in the second active register unit;

[0022] a first operation unit, configured to, in response to data update occurring in the first data branch and the second data branch, perform a multiplication operation on a data pair formed by the data in the first data branch and the second data branch, and, in response to no data update occurring in the first data branch and / or the second data branch, not perform the multiplication operation on the data in the first data branch and the second data branch;

[0023] a second operation unit, configured to accumulate the multiplication result of the first operation unit and the first multiplication-accumulation result stored in the register unit to obtain a second multiplication-accumulation result and output the result;

[0024] A storage unit is used to store the output data of the second operation unit.

[0025] Optionally, the processing unit further includes:

[0026] a control unit, configured to receive the first indication information and the second indication information, generate a target gating signal in response to receiving the first indication information and / or the second indication information, and send the target gating signal to the register unit;

[0027] The target gating signal is used to instruct the register unit not to update data in the current processing cycle.

[0028] Optionally, the first data object and the second data object are matrices, each first sub-object is a row in the first data object, and each second sub-object is a column in the second data object;

[0029] The same first data in each row of the first data object is simultaneously distributed to the respective processing units in the same row of the processing unit array in a corresponding processing cycle, and each processing unit in the same row of the processing unit array is used to process the respective first data in the same row of the first data object;

[0030] The same second data in each column of the second data object is simultaneously distributed to the respective processing units in the same column of the processing unit array in a corresponding processing cycle, and each processing unit in the same column of the processing unit array is used to process the respective second data in the same column of the second data object;

[0031] Each processing unit in the processing unit array forms a pair of data to be processed based on the corresponding first data and second data obtained, and each processing unit in the processing unit array processes its own pair of data to be processed in parallel in the same processing cycle;

[0032] The controller is configured to, when sending indication information indicating that the first data or the second data is invalid data to the processing unit corresponding to the sub-object to be processed:

[0033] Based on the broadcast mode, indication information indicating that the first data or the second data is invalid data is sent to each processing unit in the data unit array that uses the first data or the second data as one of the input data.

[0034] Optionally, when updating the primary register unit corresponding to the corresponding sub-object based at least on the current data to be processed, the controller is configured to:

[0035] The data in the standby register unit corresponding to the corresponding sub-object is updated to the current data in the main register unit corresponding to the corresponding sub-object, and the data in the main register unit corresponding to the corresponding sub-object is updated to the current data to be processed.

[0036] Optionally, when updating the primary register unit corresponding to the corresponding sub-object based at least on the current data to be processed, the controller is configured to:

[0037] determining whether the number of invalid data in each to-be-processed data corresponding to the current processing cycle of each sub-object in another data object other than the data object to which the corresponding sub-object belongs reaches a threshold; if so, not updating the standby register unit corresponding to the corresponding sub-object; if not, updating the data in the standby register unit corresponding to the corresponding sub-object with the current data in the main register unit corresponding to the corresponding sub-object;

[0038] The data in the main register unit corresponding to the corresponding sub-object is updated to the current data to be processed.

[0039] A data processing method, comprising:

[0040] Obtain a first data object and a second data object;

[0041] wherein the first data object includes a plurality of first sub-objects, the second data object includes a plurality of second sub-objects, each first sub-object and each second sub-object correspond to a corresponding pair of sub-objects to be processed, and each first data contained in the first sub-object and the corresponding second data contained in the second sub-object in the pair of sub-objects to be processed correspond to a pair of data to be processed; each pair of sub-objects to be processed is used to be assigned to a corresponding processing unit in the computing component for data processing, and the computing component includes a processing unit array formed by a plurality of processing units;

[0042] For each data pair to be processed in the sub-object pair to be processed, in response to the first data or the second data in the data pair to be processed being invalid data, indication information representing that the first data or the second data is invalid data is sent to the processing unit corresponding to the sub-object pair to be processed, so that the processing unit skips data processing of the data pair to be processed based on the indication information.

[0043] Optionally, the processing unit corresponding to each to-be-processed sub-object pair is configured to perform data processing on each to-be-processed data pair contained in the to-be-processed sub-object pair in a serial manner through a corresponding processing cycle; the processing unit processes one to-be-processed data pair in one processing cycle;

[0044] Each first sub-object in the first data object corresponds to a first primary record and a first backup record, and each second sub-object in the second data object corresponds to a second primary record and a second backup record;

[0045] The first primary record can be used to record first data to be processed by the corresponding first sub-object in the current processing cycle, and the first backup record can be used to record first data corresponding to the corresponding first sub-object in the corresponding historical processing cycle; the second primary record can be used to record second data to be processed by the corresponding second sub-object in the current processing cycle, and the second backup record can be used to record second data corresponding to the corresponding second sub-object in the corresponding historical processing cycle;

[0046] The data processing method further includes:

[0047] In response to the switching processing cycle, data update processing is performed on the corresponding register corresponding to the first sub-object, and data update processing is performed on the corresponding register corresponding to the second sub-object, and based on the data in each updated register, the first data and the second data are distributed to the corresponding processing unit to form a pair of data to be processed in the corresponding processing unit and perform data processing on the pair of data to be processed.

[0048] Optionally, the performing data updating processing on the corresponding register corresponding to the first sub-object, performing data updating processing on the corresponding register corresponding to the second sub-object, and distributing the first data and the second data to the corresponding processing unit based on the data in each updated register includes:

[0049] In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being invalid data, not updating the data of the main register and the backup register corresponding to the corresponding sub-object, not allocating the current data to be processed to the corresponding processing unit in the current processing cycle, and generating indication information indicating that the current data to be processed is invalid data, and issuing the indication information to the corresponding processing unit;

[0050] In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being valid data, updating the main register corresponding to the corresponding sub-object based on at least the current data to be processed, and distributing the data in the main register and the backup register corresponding to the corresponding sub-object to the corresponding processing unit, so that the processing unit selects required data from the data in the main register and the backup register corresponding to the corresponding sub-object;

[0051] The corresponding sub-object is the corresponding first sub-object or the second sub-object.

[0052] Optionally, the processing unit skipping data processing of the to-be-processed data pair based on the instruction information includes:

[0053] For the current processing cycle, the processing unit, based on the instruction information, does not update the first data branch corresponding to the invalid data, uses data allocated from the spare register of the corresponding sub-object to update the second data branch other than the first data branch corresponding to the invalid data, and avoids operating on the to-be-processed data pair corresponding to the invalid data;

[0054] Each processing unit corresponds to two data branches, which are respectively used to provide the processing unit with corresponding first data and second data in the processing cycle, so as to form the required data pair to be processed.

[0055] Optionally, the performing data update processing on the second data branch other than the first data branch corresponding to the invalid data using the data allocated from the spare register of the corresponding sub-object includes:

[0056] In response to the data allocated from the spare register of the corresponding sub-object being identical to the current data in the second data branch, not updating the data in the second data branch;

[0057] In response to the data allocated from the spare register of the corresponding sub-object being different from the data in the second data branch, the data in the second data branch is updated to the data allocated from the spare register of the corresponding sub-object.

[0058] Optionally, updating the master register corresponding to the corresponding sub-object based at least on the current data to be processed includes:

[0059] The data in the standby register corresponding to the corresponding sub-object is updated to the current data in the main register corresponding to the corresponding sub-object, and the data in the main register corresponding to the corresponding sub-object is updated to the current data to be processed.

[0060] Optionally, updating the master register corresponding to the corresponding sub-object based at least on the current data to be processed includes:

[0061] determining whether the number of invalid data in each to-be-processed data corresponding to a current processing cycle of each sub-object in another data object other than the data object to which the corresponding sub-object belongs reaches a threshold, and if so, not updating the standby register corresponding to the corresponding sub-object; if not, updating the data in the standby register corresponding to the corresponding sub-object with the current data in the main register corresponding to the corresponding sub-object;

[0062] The data in the main register corresponding to the corresponding sub-object is updated to the current data to be processed.

[0063] A data processing device, comprising:

[0064] An acquisition module, configured to obtain a first data object and a second data object;

[0065] The first data object includes a plurality of first sub-objects, and the second data object includes a plurality of second sub-objects. Each first sub-object and each second sub-object correspond to a corresponding sub-object pair to be processed. Each first data contained in the first sub-object and the corresponding second data contained in the second sub-object in the sub-object pair to be processed correspond to a data pair to be processed. Each sub-object pair to be processed is used to be assigned to a corresponding processing unit in the computing component for data processing.

[0066] A processing module is used to, for each data pair to be processed in the sub-object pair to be processed, in response to the first data or the second data in the data pair to be processed being invalid data, send indication information representing that the first data or the second data is invalid data to the processing unit corresponding to the sub-object pair to be processed, so that the processing unit skips data processing of the data pair to be processed based on the indication information.

[0067] An electronic device comprises any one of the above data processing chips.

[0068] A storage medium carries one or more computer instruction sets, which, when executed by an electronic device, can enable the electronic device to implement any of the above data processing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0070] Figure 1 This is a schematic diagram of the structure of a data processing chip provided by this application;

[0071] Figure 2 This is a diagram of the data processing principle of the MAC array provided by this application;

[0072] Figure 3 This is a schematic diagram provided by the present application for determining and indicating the data type at the data source;

[0073] Figure 4 This is another structural diagram of the data processing chip provided by this application;

[0074] Figure 5 This is a schematic diagram of the register units provided in the corresponding directions of the MAC array provided by the present application;

[0075] Figure 6 This is a structural diagram of the control system of each spare register unit in the row and column directions of the MAC array provided by this application;

[0076] Figure 7 This is a schematic diagram of the updated logic of the primary / backup storage unit provided by this application;

[0077] FIG8( a ) is a schematic diagram showing the increase in hardware area of ​​Solution 1 and Solution 2 in this application relative to the traditional solution;

[0078] FIG8( b ) is a schematic diagram showing the power consumption of Solution 1 and Solution 2 in this application relative to the traditional solution;

[0079] Figure 9 This is a schematic diagram of the structure of the processing unit provided by this application;

[0080] Figure 10 This is an example diagram of the PE architecture provided by this application;

[0081] Figure 11 This is a schematic diagram of controlling the data in the register Psum reg not to be updated by a target gating signal provided by the present application;

[0082] Figure 12 It is a flowchart of the data processing method provided by this application;

[0083] Figure 13 It is a schematic diagram of the structure of the data processing device provided by this application;

[0084] Figure 14 It is a schematic diagram of the composition structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0085] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0086] In neural network models, network layer weights are often quantized and pruned, resulting in a large number of zero values ​​in the weight matrix. At the same time, due to the ReLU (activation) operation, the feature map also produces a large number of zero values. For example, in typical networks such as LeNet-5, AlexNet, and VGG16, after pruning, the network layer weights can achieve a sparsity rate of more than 80%.

[0087] The applicant found that the operations in the neural network mainly include multiplication and addition (such as the multiplication and addition operations involved in the matrix multiplication of the weight matrix in the Transformer network), and the value of 0 does not contribute to the final calculation result.

[0088] Based on this, the embodiments of the present application provide a data processing method, device and data processing chip for utilizing the data sparsity characteristics in neural networks, such as the presence of a large number of zero values ​​in weights / feature maps, to optimize the power consumption of computing components such as the MAC array of the NPU, so as to improve data computing efficiency and reduce hardware resource waste.

[0089] The data processing method, device, and data processing chip provided in the embodiments of the present application can be applied to, but not limited to, electronic devices such as personal computers or servers, and can be applied to, but not limited to, natural language processing, image processing, video processing, speech recognition, industrial detection (such as equipment defect detection), and other fields.

[0090] Neural network models learn the characteristics and patterns of input information by training on large amounts of diverse data. They typically have hundreds of millions to hundreds of billions of model parameters (model parameters are variables that control the behavior of the target model) and are able to capture complex relationships and patterns in the input information. Neural network models can be generative models or generative language models (GLMs). For example, they can include large language models (LLMs), GPT (Generative Pre-trained Transformer), large visual models, large multimodal models, and so on.

[0091] The neural network model involved in the embodiments of the present application can be a general large model, or it can be an expert large model obtained after fine-tuning based on needs. The embodiments of the present application do not limit this.

[0092] The data processing chip may be, but is not limited to, an NPU. Figure 1 As shown in the composition structure diagram of the data processing chip, the data processing chip provided in the embodiment of the present application includes a computing component 10 and a controller 20.

[0093] Wherein, the computing component includes a processing unit array.

[0094] The processing unit array included in the computing component may be, but is not limited to, a MAC array. The processing units in the processing unit array may be, but are not limited to, processing elements (PEs) in the MAC array, such as MAC units.

[0095] Taking the data processing chip as an NPU as an example, the controller may be a controller in the NPU.

[0096] The controller is configured to perform the following processing steps 11)-12):

[0097] 11) Obtain a first data object and a second data object.

[0098] The first data object includes a plurality of first sub-objects, and the second data object includes a plurality of second sub-objects. Each first sub-object and each second sub-object correspond to a corresponding sub-object pair to be processed. Each first data contained in the first sub-object and the corresponding second data contained in the second sub-object in the sub-object pair to be processed correspond to a data pair to be processed. Each sub-object pair to be processed is used to be assigned to a corresponding processing unit in the computing component for data processing.

[0099] 12) For each data pair to be processed in the sub-object pair to be processed, in response to the first data or the second data in the data pair to be processed being invalid data, indication information representing that the first data or the second data is invalid data is sent to the processing unit corresponding to the sub-object pair to be processed, so that the processing unit skips data processing of the data pair to be processed based on the indication information.

[0100] The first data object and the second data object are matrices. Optionally, each first sub-object is a row in the first data object, and each second sub-object is a column in the second data object. However, this is not limiting. Each first sub-object may also be a column in the first data object, and each second sub-object may also be a row in the second data object. This embodiment of the application will primarily illustrate the solution using the example of each first sub-object being a row in the first data object and each second sub-object being a column in the second data object.

[0101] In which, when each first sub-object is a row in the first data object and each second sub-object is a column in the second data object, the same first data of each row in the first data object is simultaneously allocated to the respective processing units in the same row in the processing unit array in the corresponding processing cycle, for example, based on a parallel manner, it is simultaneously allocated to the respective processing units in the same row in the processing unit array, etc., and each processing unit in the same row in the processing unit array is used to process the first data in the same row of the first data object.

[0102] The same second data in each column of the second data object is simultaneously distributed to the processing units in the same column of the processing unit array in the corresponding processing cycle. For example, it is simultaneously distributed to the processing units in the same column of the processing unit array in a parallel manner, etc. Each processing unit in the same column of the processing unit array is used to process the second data in the same column of the second data object.

[0103] Each processing unit in the processing unit array forms a pair of data to be processed based on the corresponding first data and second data obtained, and optionally, each processing unit in the processing unit array processes its own pair of data to be processed in parallel in the same processing cycle.

[0104] The processing cycle is a data processing cycle used for data processing, and each data processing cycle is used to process a pair of data to be processed.

[0105] The data operation (such as multiplication and accumulation) tasks of each pair of data to be processed contained in the same pair of sub-objects to be processed are executed in the same processing unit.

[0106] Optionally, the processing unit corresponding to each to-be-processed sub-object pair is configured to perform data processing on each to-be-processed data pair contained in the to-be-processed sub-object pair in a serial manner through a corresponding processing cycle; the processing unit processes one to-be-processed data pair in one processing cycle.

[0107] In actual data processing scenarios, specifically based on a parallel approach, in each processing unit of the processing unit array, a moving operation can be performed on the first sub-object and the second sub-object in the corresponding pair of sub-objects to be processed according to a matching moving direction; the moving operation is used to switch the data in the first sub-object or the second sub-object in the processing unit based on the matching moving direction.

[0108] In the case where the first sub-object is a row in a matrix and the second sub-object is a column in the matrix, the moving direction matched by the first sub-object is the row direction and the moving direction matched by the second sub-object is the column direction.

[0109] In which, one move operation corresponds to one data processing cycle, and one data processing cycle is used for the processing unit to process a corresponding pair of data to be processed. By moving the first sub-object (such as the rows in matrix A) in the processing unit in the row direction and the second sub-object (such as the columns in matrix B) in the column direction in the processing unit in different data processing cycles, it is possible to achieve a pulsating manner in which the data to be processed in the first sub-object and the second sub-object are switched as needed in different data processing cycles of the same processing unit, so that the data pairs to be processed in the processing unit are continuously switched based on the switching of the data processing cycles, and each pair of data to be processed formed by position between the first sub-object and the second sub-object corresponding to the same processing unit pair is processed in sequence according to the cycle.

[0110] For example, the processing unit array is a MAC array, the first data object and the second data object are matrices A and B to be multiplied, respectively. Figure 2 The MAC array data processing principle diagram shown in the figure shows that when using the MAC array to perform matrix multiplication, each row vector in matrix A must be multiplied by each column vector in matrix B. Thus, each row vector in matrix A and each column vector in matrix B form a pair of sub-objects to be processed. Different PEs in the MAC array are responsible for performing data operations on different sub-object pairs. Data operations (such as multiplication and accumulation) for each sub-object pair within the same sub-object pair are performed on the same PE. Each PE switches the data to be processed in the corresponding sub-object pair during different data processing cycles based on the pulsating method described above, forming a pair of data to be processed during each data processing cycle. PEs in the same row of the MAC array are responsible for processing the same row of data in matrix A. The same data in the same row of matrix A will be simultaneously assigned to the PEs in the same row of the MAC array in parallel during the corresponding data processing cycle. PEs in the same column of the MAC array are responsible for processing the same column of data in matrix B. The same data in the same column of matrix B will be simultaneously assigned to the PEs in the same column of the MAC array in parallel during the corresponding data processing cycle.

[0111] The solution of the embodiment of the present application is applicable to, but not limited to, data processing scenarios of neural network models in the field of artificial intelligence, and is used to optimize the power consumption of computing components such as MAC arrays in such data processing scenarios.

[0112] In the field of artificial intelligence, neural network models primarily process data on weight matrices and feature maps using two methods: convolution and matrix multiplication. Large language models (LLMs) such as ChatGPT are currently very popular and have become one of the most important applications in the field of artificial intelligence. These large language models are generative language models based on Transformer networks. The core of the Transformer network is the attention (self-attention) mechanism, and over 90% of the computational effort in attention involves matrix multiplication. In practical applications, convolution can also be converted into matrix multiplication using corresponding conversion rules.

[0113] The neural network model can specifically perform one-dimensional convolution, two-dimensional convolution, or three-dimensional convolution on the feature map, without limitation, depending on actual needs. For example, for a one-dimensional convolution kernel of size 1×3, a one-dimensional convolution can be performed on a 1×3 feature map based on a 1×3 weight matrix. For a two-dimensional convolution kernel of size 3×3, a two-dimensional convolution can be performed on a 3×3 feature map based on a 3×3 weight matrix. These operations can all be converted into matrix-matrix multiplication operations and can all be implemented using the methods of the embodiments of the present application.

[0114] The feature map may be, but is not limited to, various types of data to be processed, such as images and voices, depending on the specific application scenario.

[0115] For data processing scenarios of neural network models in the field of artificial intelligence, for example, the first data object can be a feature map to be processed in the model network layer, the second data object can be a weight matrix of the model network layer, and the first sub-object and the second sub-object can be rows in the feature map and columns in the weight matrix, respectively. Of course, the first data object can also be the weight matrix of the model network layer, and the second data object can also be a feature map. In this case, the first sub-object and the second sub-object can be rows in the weight matrix and columns in the feature map, respectively, without limitation.

[0116] In corresponding data processing scenarios, such as data computing scenarios of neural network models, the controller in data processing chips such as NPU can specifically, but is not limited to, obtain the first data object and the second data object to be processed from the external memory of the data processing chip, or can also obtain the first data object and the second data object to be processed from its local storage data, without limitation.

[0117] Optionally, for each to-be-processed data pair in the to-be-processed sub-object pairs of the first data object and the second data object, the controller first identifies the type of data in the to-be-processed data pair before allocating the data pair to a corresponding processing unit in the processing unit array.

[0118] In actual applications, it is not limited to the controller in the data processing chip such as NPU to identify the type of data in the data pair to be processed. The type of data in the data pair to be processed can also be identified in the corresponding processor outside the data processing chip such as CPU. For the processing unit array, whether the data type identification is performed by the controller in the data processing chip or by the processor such as CPU outside the data processing chip, it can be regarded as data type identification at the data source (referred to as "data source").

[0119] The data source is a distribution source of the data pairs to be processed, and specifically can be, but is not limited to, a controller in a data processing chip such as an NPU, or a processor such as a CPU outside the data processing chip.

[0120] The data types of the data in the data pair to be processed include the data type of the first data and the data type of the second data in the data pair to be processed.

[0121] The data types include valid data types and invalid data types. The valid data type represents that the corresponding data (such as the first data or the second data) is valid data that can contribute to the calculation result in the data calculation, such as the vector of the weight matrix or the non-zero value in the feature map vector; the invalid data type represents that the corresponding data is invalid data that will not make any contribution to the calculation result in the data calculation, such as the vector of the weight matrix or the zero value in the feature map vector.

[0122] In response to the first data or the second data in the data pair to be processed being invalid data, the controller sends indication information indicating that the first data or the second data in the data pair to be processed is invalid data to the corresponding processing unit, so that the processing unit skips data processing of the data pair to be processed based on the indication information.

[0123] The processing unit corresponding to the data pair to be processed is specifically the processing unit corresponding to the sub-object pair to be processed to which the data pair to be processed belongs, and different sub-object pairs to be processed correspond to different processing units.

[0124] Among them, the processing unit skips the processing of the corresponding data pair, which means that the signal level of the multiplier, accumulator and other related operators inside the processing unit does not change, resulting in the circuit state not changing, and accordingly the multiplier, accumulator and other related operators inside the processing unit are not triggered to perform data processing, thereby skipping the data processing of the corresponding data pair.

[0125] In summary, in this embodiment, after obtaining the first data object and the second data object, the controller in the data processing chip sends, for each pair of data to be processed in the pair of sub-objects to be processed of the first data object and the second data object, an indication information representing that the first data or the second data is invalid data to the processing unit corresponding to the sub-object pair to be processed, so that the processing unit skips the data processing of the data pair to be processed based on the indication information. Thus, the present application effectively utilizes the data sparsity characteristics in data processing scenarios such as neural networks. In the case where there is invalid data in the data pair to be processed, according to the feature that invalid data such as 0 values ​​will not make any contribution to the data processing results, by sending the indication information representing that the corresponding data to be processed is invalid data to the corresponding processing unit, the processing unit skips the processing of the data pair to be processed, effectively reducing the data processing amount of the processing unit in the case of data sparsity, and correspondingly reducing the power consumption of the processing unit array such as the MAC array, thereby optimizing the power consumption of digital systems such as NPU and even SOC.

[0126] In an optional embodiment, the controller, when sending indication information indicating that the first data or the second data is invalid data to the processing unit corresponding to the sub-object to be processed, is configured to:

[0127] Based on the broadcast mode, indication information indicating that the first data or the second data is invalid data is sent to each processing unit in the data unit array that uses the first data or the second data as one of the input data.

[0128] In a matrix-matrix multiplication operation scenario, the same data to be processed in the first data object or the second data object is usually simultaneously assigned to different processing units in the processing unit array for data operations, such as different PEs in corresponding rows / columns in the MAC array.

[0129] In view of the above features, in the embodiment of the present application, a data source such as a controller determines the data type of the data to be processed and broadcasts the data type indication information of the data to be processed to the processing unit array, such as Figure 3In the example, the data type (such as 0 or non-0) of the original data to be processed is uniformly judged at the data source, and then the data type indication information is broadcast to each processing unit in the processing unit array that uses the data to be processed as one of the input data. In this way, based on the same indication information, the data type indication can be synchronously given to each processing unit in the processing unit array that uses the data to be processed as one of the input data at one time, without having to judge the data type of the input data (such as whether it is 0) separately in each processing unit in the processing unit array, thereby optimizing the data type judgment logic, avoiding each processing unit from separately judging the data type such as 0 or non-0 on the input data, improving the computing efficiency of the processing unit array and even the data processing chip such as the NPU to which it belongs, saving resources and reducing power consumption.

[0130] For the first indication information used to indicate the data type of the first data in the data pair to be processed, and the second indication information used to indicate the data type of the second data in the data pair to be processed, the first indication information can correspondingly indicate the data type of the first data to the first processing unit group in the processing unit array (such as a row of PE in the MAC array) that uses the first data as one of the input data, and the second indication information can indicate the data type of the second data to the second processing unit group in the processing unit array (such as a column of PE in the MAC array) that uses the second data as one of the input data.

[0131] In actual applications, after the data type (such as 0 or non-0) of the original data to be processed is judged at the data source, the data type indication information of the data to be processed can also be transmitted one by one to each processing unit in the processing unit array that uses the data to be processed as one of the input data according to the mapping relationship between the location identifier of the data to be processed and the processing unit identifier in the processing unit array based on a non-broadcast method, such as based on a wired transmission method. This implementation method can also optimize the data type judgment logic of the processing unit array, avoiding each processing unit from individually judging the data type such as 0 or non-0 on the input data, thereby improving the computing efficiency of the processing unit array and even the data processing chips such as the NPU to which it belongs, saving resources and reducing power consumption.

[0132] In an alternative embodiment, see Figure 4 As shown in the composition structure diagram of the data processing chip, the data processing chip in the present application further includes a first main register unit group 30, a first backup register unit group 40, a second main register unit group 50, and a second backup register unit group 60.

[0133] Wherein, each processing unit group in the first direction of the processing unit array corresponds to a first main register unit and a first backup register unit, and is used to process a first sub-object in the first data object based on the corresponding first main register unit and first backup register unit; each processing unit group in the second direction of the processing unit array corresponds to a second main register unit and a second backup register unit, and is used to process a second sub-object in the second data object based on the corresponding second main register unit and second backup register unit.

[0134] If the first sub-object is a row in the matrix (first data object) to which it belongs, and the second sub-object is a column in the matrix (second data object) to which it belongs, then the first direction is the row direction and the second direction is the column direction. If the first sub-object is a column in the matrix (first data object) to which it belongs, and the second sub-object is a row in the matrix (second data object) to which it belongs, then the first direction is the column direction and the second direction is the row direction. The embodiments of this application will mainly use the first case, i.e., the first direction is the row direction and the second direction is the column direction, as an example to illustrate the solution.

[0135] The first primary register unit can be used to store first data to be processed by the corresponding first sub-object in the current processing cycle, and the first backup register unit can be used to store first data corresponding to the corresponding first sub-object in the corresponding historical processing cycle. The second primary register unit can be used to store second data to be processed by the corresponding second sub-object in the current processing cycle, and the second backup register unit can be used to store second data corresponding to the corresponding second sub-object in the corresponding historical processing cycle.

[0136] Take the MAC array as an example, see Figure 5 A schematic diagram of register units arranged in corresponding directions of the MAC array is provided. Each PE in the same row of the MAC array corresponds to a first active register unit Rowi_ff and a first standby register unit Rowi_ff2; each PE in the same column of the MAC array corresponds to a second active register unit colj_ff and a second standby register unit colj_ff2.

[0137] Here, the i represents the row identifier of the row corresponding to Rowi_ff or Rowi_ff2 in the MAC array, and the j represents the column identifier of the column corresponding to colj_ff or colj_ff2 in the MAC array.

[0138] Assuming that the first data object and the second data object are matrices A and B, respectively, and the MAC array needs to perform a multiplication operation on matrices A and B, the first primary register unit Rowi_ff can be used to store the first data to be processed in the i-th row of matrix A during the current data processing cycle, and the first backup register unit Rowi_ff2 can be used to store the first data corresponding to the i-th row of matrix A during the corresponding historical processing cycle. The second primary register unit colj_ff can be used to store the second data to be processed in the j-th column of matrix B during the current data processing cycle, and the second backup register unit colj_ff2 can be used to store the second data corresponding to the j-th column of matrix B during the corresponding historical processing cycle.

[0139] Each PE in the i-th row of the MAC array is used to process the i-th row of matrix A based on the corresponding first active register unit Rowi_ff and the first backup register unit Rowi_ff2; each PE in the j-th column of the MAC array is used to process the j-th column of matrix B based on the corresponding second active register unit colj_ff and the second backup register unit colj_ff2.

[0140] In practice, each register unit can be set at the data source end, such as the controller end of the data processing chip, and the corresponding relationship between each register unit and the processing unit in the processing unit array is maintained at the data source end.

[0141] This embodiment provides primary and backup register units corresponding to processing unit groups in different directions of the processing unit array at the data source end of the processing unit array. This facilitates the controller to perform data allocation or input control to the processing units in different directions of the processing unit array based on the data storage functions provided by the corresponding primary / backup register units, so as to match the data types of the data to be processed in different directions (row / column directions) of the processing units, thereby providing support for the processing units to skip data processing of data pairs to be processed that contain invalid data.

[0142] In an optional embodiment, the controller in the data processing chip is also used to: in response to the switching processing cycle, perform data update processing on the corresponding register unit corresponding to the first sub-object, perform data update processing on the corresponding register unit corresponding to the second sub-object, and distribute the first data and the second data to the corresponding processing unit based on the data in each register unit after the update, so as to form a pair of data to be processed in the corresponding processing unit and perform data processing on the pair of data to be processed.

[0143] The processing cycle is a data processing cycle used for data processing, and different data processing cycles are used to process different pairs of data to be processed.

[0144] In response to the switching processing cycle, the controller performs the above-mentioned data updating processing on each register unit and the data distribution processing to the processing unit based on the data updating processing, which can be further implemented as the following steps 21)-22):

[0145] 21) In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being invalid data, not updating data on the main register unit and the backup register unit corresponding to the corresponding sub-object, not allocating the current data to be processed to the corresponding processing unit in the current processing cycle, and generating indication information indicating that the current data to be processed is invalid data, so as to send the indication information to the corresponding processing unit.

[0146] The corresponding sub-object is the corresponding first sub-object or the corresponding second sub-object. For example, it is the corresponding row in the first data object such as a feature map, or the corresponding column in the second data object such as a weight matrix, depending on the data type of the current to-be-processed data of the first sub-object and the second sub-object in the to-be-processed sub-object pair.

[0147] Each processing unit corresponds to two paths of input data, specifically the first data to be processed by the first sub-object and the second data to be processed by the second sub-object in the corresponding pair of sub-objects to be processed.

[0148] Among them, if in the current processing cycle switched to, the first data currently to be processed by the first sub-object is invalid data, the data of the first main registration unit and the first backup registration unit corresponding to the first sub-object will not be updated, and the first data will not be allocated to the processing unit corresponding to the first sub-object in the current processing cycle, and a first indication information indicating that the first data is invalid data is sent to the processing unit, for example, by broadcasting, to each processing unit in the processing unit array that uses the first data as one of the input data.

[0149] Similarly, if in the current processing cycle switched to, the second data currently to be processed by the second sub-object is invalid data, the data of the second main registration unit and the second backup registration unit corresponding to the second sub-object will not be updated, and the second data will not be allocated to the processing unit corresponding to the second sub-object in the current processing cycle, and a second indication information indicating that the second data is invalid data is sent to the processing unit, for example, by broadcasting to each processing unit in the processing unit array that uses the second data as one of the input data.

[0150] For example, in Figure 5In the data processing scenario, for the row-column pair to be processed formed by the i-th row in matrix A and the j-th column in matrix B, if the current data to be processed in the i-th row of matrix A is 0 in the current processing cycle switched to, Rowi_ff and Rowi_ff2 are not updated, and the data is not allocated to the i-th row PE of the MAC array, and a first indication information such as broadcast indicating that the data is 0 is sent to the i-th row PE of the MAC array; similarly, if the current data to be processed in the j-th column of matrix B is 0 in the current processing cycle switched to, colj_ff and colj_ff2 are not updated, and the data is not allocated to the j-th column PE of the MAC array, and a second indication information such as broadcast indicating that the data is 0 is sent to the j-th column PE of the MAC array.

[0151] In this embodiment, when the current processing cycle switched to is invalid data corresponding to the corresponding sub-object, the controller does not update the data of the main and backup register units corresponding to the corresponding sub-object, that is, maintains the existing data of the main and backup register units corresponding to the corresponding sub-object, and does not allocate the current data to the corresponding processing unit in the current processing cycle. This is used to prevent the input of the branch corresponding to the current data to be processed (invalid data) of the corresponding processing unit from changing, and then based on the input unchanged event and the received indication information, the processing unit does not flip in the current processing cycle and does not perform data operations, so as to skip the processing of the data pair to be processed to which the invalid data belongs, thereby avoiding invalid power consumption.

[0152] The flip in the embodiment of the present application refers to a device such as an arithmetic unit or a register in a processing unit triggering / starting a data operation or data update.

[0153] 22) In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being valid data, updating the primary register unit corresponding to the corresponding sub-object based at least on the current data to be processed, and allocating the data in the primary register unit and the backup register unit corresponding to the corresponding sub-object to the corresponding processing unit, so that the processing unit selects required data from the data in the primary register unit and the backup register unit corresponding to the corresponding sub-object.

[0154] Among them, if in the current processing cycle switched to, the first data currently to be processed by the first sub-object is valid data, then the first main register unit corresponding to the first sub-object is updated based on at least the first data, and the first backup register unit is updated or not updated, and the data in the first main register unit and the first backup register unit corresponding to the first sub-object are distributed to the processing unit with the first data as one of the input data, so that the processing unit with the first data as one of the input data selects the required data from the data of the first main register unit and the first backup register unit as the input data of its corresponding branch.

[0155] Similarly, if in the current processing cycle switched to, the second data currently to be processed by the second sub-object is valid data, then the second main register unit corresponding to the second sub-object is updated at least based on the second data, and the second backup register unit is updated or not updated, and the data in the second main register unit and the second backup register unit corresponding to the second sub-object are distributed to the processing unit that uses the second data as one of the input data, so that the processing unit that uses the second data as one of the input data selects the required data from the data of the second main register unit and the second backup register unit as the input data of its corresponding branch.

[0156] For example, in Figure 5 In the data processing scenario, for the row-column pair to be processed formed by the i-th row of matrix A and the j-th column of matrix B, if the current data to be processed of the i-th row of matrix A is non-zero in the current processing cycle switched to, at least Rowi_ff is updated, and the data in Rowi_ff and Rowi_ff2 are allocated to the i-th row PE of the MAC array, so that the i-th row PE of the MAC array selects the required data from the data in Rowi_ff and Rowi_ff2 as the input data of its corresponding branch; if the current data to be processed of the j-th column of matrix B is non-zero in the current processing cycle switched to, at least colj_ff is updated, and the data in colj_ff and colj_ff2 are allocated to the j-th column PE of the MAC array, so that the j-th column PE of the MAC array selects the required data from the data in colj_ff and colj_ff2 as the input data of its corresponding branch.

[0157] In the current data processing cycle, if one path of data in the processing unit is invalid data, the data in the processing unit will not be updated, that is, the input of the data will not change, and the data in the corresponding standby register unit will be selected for the other path as the input data of the other path. Based on the fact that at least one path of data input has not changed, the operators inside the processing unit will not flip during the current processing cycle, that is, the signal level of the operator will not change, and the data operation will not be performed accordingly, and the processing of the data pair to be processed in the current processing cycle will be skipped to avoid invalid power consumption.

[0158] If both data of the processing unit are invalid data, then both data of the processing unit will not be updated, that is, the inputs of the two data will not change. Accordingly, based on the fact that the two inputs do not change, the internal operators of the processing unit will not flip in the current processing cycle, thereby skipping the processing of the data pair to be processed in the current processing cycle, and avoiding invalid power consumption.

[0159] If both data of the processing unit are valid data, the two data of the processing unit select the data in the main register unit on the corresponding branch, and the two data inputs are updated based on the data in the corresponding main register unit, which causes the various operators inside the processing unit to flip in the current processing cycle to process the data pair to be processed formed by the two valid data to be processed, such as performing the multiplication and accumulation operations involved in the matrix multiplication operation on the valid data pair to be processed.

[0160] To summarize, this embodiment performs data update processing on the corresponding main / backup register units and data distribution processing to the processing units based on the data type of the data to be processed in the controller of the data processing chip, and sends indication information to the processing unit when necessary to indicate that the corresponding data to be processed is invalid data, so that the processing unit can skip the processing of the data pairs to be processed containing invalid data. Therefore, this application effectively utilizes the data sparsity characteristics in data processing scenarios such as neural networks, and performs corresponding data update and distribution control in the controller based on this characteristic, thereby avoiding the operation of invalid data in the processing unit, and correspondingly avoiding the invalid power consumption of the processing unit, thereby achieving the optimization of the power consumption of digital systems such as NPU and even SOC.

[0161] In an optional embodiment, in response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being valid data, the controller updates the main register unit corresponding to the corresponding sub-object based at least on the current data to be processed, for:

[0162] The data in the standby register unit corresponding to the corresponding sub-object is updated to the current data in the main register unit corresponding to the corresponding sub-object, and the data in the main register unit corresponding to the corresponding sub-object is updated to the current data to be processed.

[0163] If the first data currently to be processed in the first sub-object is valid data during the current processing cycle, the data in the first backup register unit corresponding to the first sub-object is updated with the data in the first primary register unit corresponding to the first sub-object, and the data in the first primary register unit corresponding to the first sub-object is updated with the first data currently to be processed. For example, if the data currently to be processed in the i-th row of matrix A is a non-zero value, the data in Rowi_ff2 is updated with the data in Rowi_ff, and the data in Rowi_ff is updated with the data currently to be processed in the i-th row of matrix A.

[0164] Similarly, if the second data currently to be processed in the second sub-object is valid data during the current processing cycle, the data in the second backup register unit corresponding to the second sub-object is updated with the data in the second primary register unit corresponding to the second sub-object, and the data in the second primary register unit corresponding to the second sub-object is updated with the second data currently to be processed. For example, if the data currently to be processed in the j-th column of matrix B is a non-zero value, the data in colj_ff2 is updated with the data in colj_ff, and the data in colj_ff is updated with the data currently to be processed in the j-th column of matrix B.

[0165] In this embodiment, the data in the standby register unit corresponding to each direction (row direction, column direction) of the processing unit array is always one data processing cycle later than the data in the main register unit corresponding to the direction.

[0166] To summarize, when the data to be processed on a certain path of the processing unit is invalid data such as a value of 0, the controller does not distribute data to the processing unit based on the primary / backup register unit corresponding to the path, but only broadcasts to the processing unit an indication information indicating that the current data to be processed on the path is invalid data. When the data to be processed on the path is valid data such as a value other than 0, the controller distributes data to the processing unit based on the data in the primary register unit and the backup register unit corresponding to the path, so that the processing unit can control its own data input, operator flipping, etc. based on the reception of the corresponding indication information and the data to be processed. When one path of data in the processing unit is invalid data, the data on the path is not updated, and the other path of data selects the data in the corresponding backup register unit, for example, the data of the previous data processing cycle stored in the backup register unit is selected. In this way, both input data are not updated, and the operator in the processing unit does not flip in the current data processing cycle, thereby skipping the processing of the pair of data to be processed containing invalid data, thereby avoiding invalid power consumption.

[0167] In an optional embodiment, in response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being valid data, the controller updates the main register unit corresponding to the corresponding sub-object based on at least the current data to be processed, and is configured to perform the following steps 31)-32):

[0168] 31) Determine whether the number of invalid data in each to-be-processed data corresponding to each sub-object in another data object other than the data object to which the corresponding sub-object belongs in the current processing cycle reaches a threshold; if so, do not update the backup register unit corresponding to the corresponding sub-object; if not, update the data in the backup register unit corresponding to the corresponding sub-object to the current data in the primary register unit corresponding to the corresponding sub-object.

[0169] Among them, if in the current processing cycle switched to, the first data currently to be processed in the first sub-object is valid data, then it is determined whether the number of invalid data in each second data to be processed corresponding to each second sub-object in the second data object in the current processing cycle reaches a threshold; if the threshold is reached, the first backup registration unit corresponding to the first sub-object is not updated, that is, the existing data in the first backup registration unit is maintained; if the threshold is not reached, the data in the first backup registration unit is updated to the current data in the first main registration unit corresponding to the first sub-object.

[0170] Similarly, if in the current processing cycle switched to, the second data currently to be processed in the second sub-object is valid data, then it is determined whether the number of invalid data in the respective first data to be processed corresponding to each first sub-object in the first data object in the current processing cycle reaches a threshold; if the threshold is reached, the second standby registration unit corresponding to the second sub-object is not updated, that is, the existing data in the second standby registration unit is maintained; if the threshold is not reached, the data in the second standby registration unit is updated to the current data in the second main registration unit corresponding to the second sub-object.

[0171] For example, for Figure 5 In a data processing scenario, assuming that matrices A and B to be multiplied are 32×32 matrices, each first data point x in matrix A must be processed with the data points y in the corresponding positions in the 32 columns of matrix B to form a pair for data operation. That is, each x must be operated on separately with the corresponding 32 y points. Based on the solution of this embodiment, if the number of invalid data points, such as 0s, in the 32 y points corresponding to x reaches a threshold, the first standby register unit x_ff2 corresponding to x is not updated. Otherwise, x_ff2 is updated. The update logic of the first standby register unit x_ff2 corresponding to x is specifically shown in Table 1.

[0172] Table 1

[0173]

[0174] In Table 1, x_cur represents a first data currently to be processed in matrix A, and the 32 y_cur represent the second data corresponding to x_cur in the 32 columns of matrix B. "More" and "Less" in Table 1 respectively indicate that the number of zero values ​​in the 32 y_cur reaches or does not reach the threshold, and "Y" and "N" respectively indicate whether the first spare register unit x_ff2 corresponding to x_cur is updated or not updated.

[0175] Similarly, each second data y in matrix B needs to form a pair of data to be processed with the data x at the corresponding position in the 32 rows of matrix A for data operation. That is, each y needs to be operated with the corresponding 32 xs respectively.

[0176] Based on the solution of this embodiment, if the number of invalid data such as 0 in the 32 x corresponding to y reaches a threshold, the second spare register unit y_ff2 corresponding to y is not updated, otherwise y_ff2 is updated. The update logic of the second spare register unit y_ff2 corresponding to y is shown in Table 2.

[0177] Table 2

[0178]

[0179] In Table 2, y_cur represents a second data currently to be processed in matrix B, and the 32 x_cur represent the first data corresponding to y_cur in the 32 rows of matrix A. "More" and "Less" in Table 2 respectively indicate that the number of zero values ​​in the 32 x_cur reaches or does not reach the threshold, and "Y" and "N" respectively indicate whether the second spare register unit y_ff2 corresponding to y_cur is updated or not updated.

[0180] In this embodiment, the main register unit in the row / column direction is updated when the data to be processed in the data processing cycle currently switched to in the corresponding direction is valid data, and is not updated when it is invalid data; the backup register unit in the row / column direction is updated only when the number of valid data to be processed in the current data processing cycle in the other direction other than the corresponding direction reaches a threshold, otherwise it is not updated.

[0181] See also Figure 6 , showing Figure 5In a MAC array data processing scenario, the control system structure and logic of each spare register of the MAC array in the row and column directions, wherein in the current data processing cycle switched to, if the current to-be-processed data xi in the i-th row of matrix A is not 0, and the number of 0 values ​​in each y corresponding to each column of matrix B is less than the threshold value (y_0_cnt<threshold), then the first spare register Rowi_ff2 corresponding to the data is updated; otherwise, it is not updated; if the current to-be-processed data yj in the j-th column of matrix B is not 0, and the number of 0 values ​​in each x corresponding to each row of matrix A is less than the threshold value (x_0_cnt<threshold), then the second spare register colj_ff2 corresponding to the data is updated; otherwise, it is not updated.

[0182] During implementation, optionally, when switching to the current data processing cycle, the number of 0 values ​​in the current data to be processed in the row and column directions can be counted respectively, and then compared with the threshold value, and based on the comparison result, an enable signal is generated to indicate whether the backup register in the other direction is updated, and the enable signal is broadcast to all backup registers in the other direction to realize the update control of the backup register.

[0183] See also Figure 7 The backup register in a certain direction will be updated only when the number of 0 values ​​in the current data to be processed in the other direction is less than the threshold and the data of this path (i.e. the data branch in the direction corresponding to the backup register) in the current data processing cycle is not 0. Figure 7 As shown, a schematic diagram of the update logic of the first main register unit Row0_ff and the first backup register unit Row0_ff2 in the row direction is provided, wherein En=x_not_0 indicates that when the row direction data to be processed x is non-zero in the data processing cycle currently switched to, an enable signal is generated to update Row0_ff, specifically updating the data in Row0_ff to x; En=x_not_0&(y_0_cnt<threshold) indicates that when the row direction data to be processed x is non-zero in the data processing cycle currently switched to and the number of 0 values ​​in each column direction data to be processed y corresponding to x is less than the threshold threshold, an enable signal is generated to update Row0_ff2, specifically updating the data in Row0_ff2 to the data before the update in Row0_ff.

[0184] 32) The data in the main register corresponding to the corresponding sub-object is updated to the current data to be processed.

[0185] If the first data currently to be processed in the first sub-object is valid data during the current processing cycle, the data in the first main register corresponding to the first sub-object is updated with the first data currently to be processed. For example, if the data currently to be processed in the i-th row of matrix A is a non-zero value, the data in Rowi_ff is updated with the data currently to be processed in the i-th row of matrix A.

[0186] Similarly, if the second data currently to be processed in the second sub-object is valid data during the current processing cycle, the data in the second main register corresponding to the second sub-object is updated with the second data currently to be processed. For example, if the data currently to be processed in the j-th column of matrix B is a non-zero value, the data in colj_ff is updated with the data currently to be processed in the j-th column of matrix B.

[0187] In the scheme of the previous embodiment, the data in the backup register corresponding to each direction (row direction, column direction) of the processing unit array is always one data processing cycle later than the data in the main register corresponding to that direction. In this embodiment, if the same processing unit has invalid data such as 0 in two or more consecutive data processing cycles in a certain branch, the other branch will select the data in the previous data processing cycle of the cycle in each cycle of the consecutive data processing cycle when the processing unit operates. However, in fact, the data in the consecutive data processing cycles of the other branch is very likely different. This will cause the input data of the other branch to change between different cycles of the consecutive data processing cycle when the consecutive data processing cycle of the branch is 0 or other invalid data, thereby causing the data input-related devices inside the PE (such as registers for storing input data) to still have data flips, thereby resulting in invalid power consumption.

[0188] To address this issue, unlike the solution of the previous embodiment, in the solution of this embodiment, whether the standby register unit is updated is determined based on the number of invalid data in the data to be processed in the direction other than the direction corresponding to the standby register unit. If the number of invalid data such as 0 values ​​in the current data to be processed in the other direction is large and reaches a threshold, it indicates that in the processing unit group corresponding to the standby register unit (such as Rowi_ff2 in the i-th row PE of the MAC array), most processing units need to use the data in the standby register unit and not the data in the main register unit. Accordingly, the data in the standby register unit may not be updated. In this way, when a processing unit encounters invalid data in multiple consecutive data processing cycles in one branch, the other branch also (as much as possible) avoids data updating, thereby preventing data input-related devices (such as registers for storing input data) from generating data flips, thereby reducing invalid power consumption.

[0189] The comparison results of the two solutions provided by the above two embodiments of the present application and the traditional solution are shown in Figure 8(a) and Figure 8(b), respectively. Figure 8(a) shows the increase in hardware area of ​​the two solutions of Solution 1 and Solution 2 in the present application relative to the traditional solution, and Figure 8(b) shows the power consumption of the two solutions of Solution 1 and Solution 2 in the present application relative to the traditional solution. Solution 1 and Solution 2 represent the solution of the previous embodiment of the present application and the solution of the present embodiment, respectively.

[0190] Solution 1, Solution 2, and the traditional solution in this application are represented as spare_1, spare_n, and normal in Figures 8(a) and 8(b), respectively. The comparison results shown in Figures 8(a) and 8(b) show that compared to Solution 1 (spare_1), Solution 2 (spare_n) has a larger hardware area. However, after the data of the data objects to be processed (such as feature maps / weight matrices) reaches a certain sparsity (such as around 25%), Solution 2 has better power consumption optimization effects. When the sparsity is low (such as sparsity <25%), Solution 2 has higher power consumption due to the addition of additional logic. When the sparsity is high (such as sparsity >40%), Solution 2 has the best power consumption, outperforming the traditional solution (normal) and Solution 1. The higher the sparsity, the more significant the optimization effect. In different application systems, the sparsity of data objects will vary. During implementation, it can be decided in advance whether to adopt the solution of the embodiment of this application and which of Solution 1 and Solution 2 to adopt based on the sparsity of the system scenario. In high-sparseness scenarios, this application can bring obvious power consumption optimization effects compared with traditional solutions.

[0191] In an alternative embodiment, see Figure 9 As shown in the processing unit composition structure diagram, the processing units in the processing unit array include a first receiving unit 901, a second receiving unit 902, a first operating unit 903, a second operating unit 904 and a register unit 905.

[0192] Among them, the first receiving unit is used to receive the first data allocated by the controller based on the corresponding first main register unit and the first backup register unit, and the second indication information indicating that the second data to be processed in the current processing cycle is invalid data; and in response to not receiving the first data, not updating the data of the first data branch; in response to receiving the first data and the second indication information, updating the data of the first data branch based on the first data in the first backup register unit; in response to receiving the first data and not receiving the second indication information, updating the data in the first data branch based on the first data in the first main register unit.

[0193] The second receiving unit is configured to receive second data allocated by the controller based on the corresponding second main register unit and second backup register unit, and first indication information indicating that the first data to be processed in the current processing cycle is invalid data; and in response to not receiving the second data, not updating the data on the second data branch; in response to receiving the second data and the first indication information, updating the data on the second data branch based on the second data in the second backup register unit; and in response to receiving the second data and not receiving the first indication information, updating the data in the second data branch based on the second data in the second main register unit.

[0194] The two receiving units, namely the first receiving unit and the second receiving unit, can be two selectors with corresponding data interfaces. Specifically, two selectors can be set at the input end of the processing unit, such as the input end of the PE in the MAC array, as the first receiving unit and the second receiving unit.

[0195] Each selector selects the value of the current path data it is responsible for based on the data type of the other path data received. If the data type of the other path data indicates that the other path data is invalid data such as 0, the data in the backup register unit corresponding to the current path is selected as the input data for the current path. If the data type of the other path data indicates that the other path data is valid data such as non-0, the data in the main register unit corresponding to the current path is selected as the input data for the current path. By selecting the data in the backup register unit corresponding to the current path as the input data for the current path when the other path data is invalid data such as 0, updates to the current path input data are minimized, and accordingly, flipping of the current path due to input data updates is minimized, thereby reducing ineffective power consumption.

[0196] The first operation unit is configured to perform a multiplication operation on a data pair formed by the data in the first data branch and the second data branch in response to data update in the first data branch and the second data branch, and not perform a multiplication operation on the data in the first data branch and the second data branch in response to no data update in the first data branch and / or the second data branch.

[0197] The second operation unit is used to accumulate the multiplication result of the first operation unit and the first multiplication and accumulation result stored in the storage unit to obtain a second multiplication and accumulation result and output it.

[0198] A storage unit is used to store the output data of the second operation unit.

[0199] The first operation unit and the second operation unit may be a multiplier and an adder respectively, and the storage unit may be a register.

[0200] Take the PE in the MAC array as an example, see Figure 10 The example diagram of the PE architecture shown includes a multiplier Mul for multiplication operations, an accumulator Acc for accumulation operations, and a register Psumreg for storing the multiplication and accumulation results corresponding to each pair of data to be processed that has completed the calculation. In addition, two selectors Mux1 and Mux2 are provided at the PE input end. Each selector is used to receive the current path data it is responsible for (including the data in the corresponding main register and backup register), as well as data type indication information of another path data other than the current path data it is responsible for. Based on the data type indication information of the other path data, whether the other path data is invalid data such as 0, the selector selects the data in the backup register corresponding to the current path as the current path input data or selects the data in the main register corresponding to the current path as the current path input data.

[0201] Figure 10 In the example, "x_is_0" is an indication signal indicating that the current data to be processed in the first data object, such as the row vector of matrix A, is 0, and "y_is_0" is an indication signal indicating that the current data to be processed in the second data object, such as the column vector of matrix B, is 0. The two input arrows corresponding to each selector represent the two optional data provided by the main register and the backup register corresponding to the selector, respectively. If a selector receives an indication signal indicating that the data of another path is 0, the selector selects the data in its corresponding backup register as the input data of this path, for example Figure 10 When Mux1 receives the "y_is_0" indication signal, it selects the data row_x_ff2 in the corresponding standby register as the input data for this channel. Otherwise, if it does not receive the "y_is_0" indication signal, it selects the data row_x in the corresponding main register as the input data for this channel. Mux2 works in the same way.

[0202] In this embodiment, in response to receiving at least one channel of invalid data inside a processing unit such as a PE, the arithmetic unit (such as a multiplier and an adder) inside the processing unit does not flip in the current processing cycle and does not perform data operations, so as to skip processing the pair of data to be processed corresponding to the invalid data, thereby avoiding invalid power consumption, thereby effectively reducing the data processing capacity of the processing unit in the case of data sparseness, and correspondingly reducing the power consumption of processing unit arrays such as MAC arrays, thereby optimizing the power consumption of digital systems such as NPUs and even SOCs.

[0203] In an optional embodiment, the processing unit in the data processing chip may further include:

[0204] A control unit is configured to receive the first indication information and the second indication information, generate a target gating signal in response to receiving the first indication information and / or the second indication information, and send the target gating signal to the register unit.

[0205] Optionally, the control unit may be a door control unit.

[0206] The target gating signal is used to instruct the register unit not to update data in the current processing cycle. The target gating signal may be a clock gating signal.

[0207] If the control unit receives the first indication information and / or the second indication information, indicating that at least one data channel of the processing unit is currently invalid data, the processing unit will not receive the invalid data based on the processing method provided above, and will not trigger the processing of the data pair to be processed corresponding to the invalid data (i.e., skip the processing of the data pair to be processed corresponding to the invalid data), so that the current data processing cycle will not generate the operation result of the data pair to be processed corresponding to the invalid data, and accordingly, there is no need to update the data in the register unit, that is, there is no need to update the multiplication and accumulation results corresponding to each data pair to be processed that has completed the calculation and stored in Psum reg.

[0208] Based on this, in this case, the present embodiment utilizes the control unit to generate a target gating signal and sends the target gating signal to the register unit. In response to receiving the signal, the register unit may not update the data in the register unit based on the operation result of the adder, or may directly maintain the data in the register unit unchanged.

[0209] See also Figure 11 , which shows a schematic diagram of controlling the data in the register Psumreg not to be updated by the target gating signal when invalid data exists in two data paths in the PE of the MAC array.

[0210] Wherein, En=~(x_is_0|y_is_0) represents the target gating signal initiated by the gating unit, such as the clockgating signal, which is used to control the register Psum reg not to flip in the current data processing cycle so that the data in Psumreg is not updated in the current data processing cycle.

[0211] This embodiment provides a control unit in the processing unit, and enables the control unit to generate a target gating signal for instructing the register unit not to update data in the current processing cycle in response to receiving first indication information indicating that first data to be processed by the processing unit is invalid data and / or receiving second indication information indicating that second data to be processed by the processing unit is invalid data. This can effectively prevent the register unit from flipping in the current processing cycle when invalid data exists, thereby further reducing the power consumption of the processing unit and achieving further optimization of the power consumption of digital systems such as NPUs and even SOCs.

[0212] The present application also provides a data processing method. Figure 12 The method flow diagram shown in FIG. 1 shows a method flow diagram of a data processing method, wherein the data processing method includes the following processing steps:

[0213] Step 1201: Obtain a first data object and a second data object; wherein the first data object includes multiple first sub-objects, the second data object includes multiple second sub-objects, each first sub-object and each second sub-object correspond to a corresponding sub-object pair to be processed, and each first data contained in the first sub-object and the corresponding second data contained in the second sub-object in the sub-object pair to be processed correspond to a data pair to be processed.

[0214] Each to-be-processed sub-object is used to perform data processing on a corresponding processing unit assigned to a computing component, wherein the computing component includes a processing unit array formed by a plurality of processing units;

[0215] Step 1202: For each data pair to be processed in the sub-object pair to be processed, in response to the first data or the second data in the data pair to be processed being invalid data, an indication information representing that the first data or the second data is invalid data is sent to the processing unit corresponding to the sub-object pair to be processed, so that the processing unit skips data processing of the data pair to be processed based on the indication information.

[0216] In an optional embodiment, the processing unit corresponding to each pair of sub-objects to be processed is configured to perform data processing on each pair of data to be processed contained in the pair of sub-objects to be processed in a serial manner through a corresponding processing cycle; the processing unit processes one pair of data to be processed in one processing cycle;

[0217] Each first sub-object in the first data object corresponds to a first primary record and a first backup record, and each second sub-object in the second data object corresponds to a second primary record and a second backup record;

[0218] The first main record can be used to record the first data to be processed by the corresponding first sub-object in the current processing cycle, and the first backup record can be used to record the first data corresponding to the corresponding first sub-object in the corresponding historical processing cycle; the second main record can be used to record the second data to be processed by the corresponding second sub-object in the current processing cycle, and the second backup record can be used to record the second data corresponding to the corresponding second sub-object in the corresponding historical processing cycle.

[0219] The data processing method further includes:

[0220] In response to the switching processing cycle, data update processing is performed on the corresponding register corresponding to the first sub-object, and data update processing is performed on the corresponding register corresponding to the second sub-object, and based on the data in each updated register, the first data and the second data are distributed to the corresponding processing unit to form a pair of data to be processed in the corresponding processing unit and perform data processing on the pair of data to be processed.

[0221] The first primary record and the first backup record corresponding to the first sub-object may be the data recorded in the first primary registration unit and the first backup registration unit corresponding to the first sub-object, respectively; similarly, the second primary record and the second backup record corresponding to the second sub-object may be the data recorded in the second primary registration unit and the second backup registration unit corresponding to the second sub-object, respectively.

[0222] In an optional embodiment, the performing data updating processing on the corresponding register corresponding to the first sub-object, the performing data updating processing on the corresponding register corresponding to the second sub-object, and distributing the first data and the second data to the corresponding processing unit based on the updated data in each register, includes:

[0223] In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being invalid data, not updating the data of the main register and the backup register corresponding to the corresponding sub-object, not allocating the current data to be processed to the corresponding processing unit in the current processing cycle, and generating indication information indicating that the current data to be processed is invalid data, and issuing the indication information to the corresponding processing unit;

[0224] In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being valid data, updating the main register corresponding to the corresponding sub-object based on at least the current data to be processed, and distributing the data in the main register and the backup register corresponding to the corresponding sub-object to the corresponding processing unit, so that the processing unit selects required data from the data in the main register and the backup register corresponding to the corresponding sub-object;

[0225] The corresponding sub-object is the corresponding first sub-object or the second sub-object.

[0226] In an optional implementation manner, the processing unit skips data processing of the to-be-processed data pair based on the instruction information, including:

[0227] For the current processing cycle, the processing unit, based on the instruction information, does not update the first data branch corresponding to the invalid data, uses data allocated from the spare register of the corresponding sub-object to update the second data branch other than the first data branch corresponding to the invalid data, and avoids operating on the to-be-processed data pair corresponding to the invalid data;

[0228] Each processing unit corresponds to two data branches, which are respectively used to provide the processing unit with corresponding first data and second data in the processing cycle, so as to form the required data pair to be processed.

[0229] In an optional embodiment, the step of performing data updating processing on the second data branch other than the first data branch corresponding to the invalid data using the data allocated from the spare register of the corresponding sub-object includes:

[0230] In response to the data allocated from the spare register of the corresponding sub-object being identical to the current data in the second data branch, not updating the data in the second data branch;

[0231] In response to the data allocated from the spare register of the corresponding sub-object being different from the data in the second data branch, the data in the second data branch is updated to the data allocated from the spare register of the corresponding sub-object.

[0232] In an optional implementation manner, updating the master register corresponding to the corresponding sub-object based at least on the current data to be processed includes:

[0233] The data in the standby register corresponding to the corresponding sub-object is updated to the current data in the main register corresponding to the corresponding sub-object, and the data in the main register corresponding to the corresponding sub-object is updated to the current data to be processed.

[0234] In an optional implementation manner, updating the master register corresponding to the corresponding sub-object based at least on the current data to be processed includes:

[0235] determining whether the number of invalid data in each to-be-processed data corresponding to a current processing cycle of each sub-object in another data object other than the data object to which the corresponding sub-object belongs reaches a threshold, and if so, not updating the standby register corresponding to the corresponding sub-object; if not, updating the data in the standby register corresponding to the corresponding sub-object with the current data in the main register corresponding to the corresponding sub-object;

[0236] The data in the main register corresponding to the corresponding sub-object is updated to the current data to be processed.

[0237] The data processing method provided in the embodiment of the present application corresponds to the data processing chip disclosed above, so the description of the data processing method is relatively simple. For relevant similarities, please refer to the description of the data processing chip above, which will not be repeated here.

[0238] The present application also provides a data processing device, the structure of which is as follows: Figure 13 As shown, including:

[0239] The acquisition module 1301 is configured to obtain a first data object and a second data object.

[0240] The first data object includes a plurality of first sub-objects, and the second data object includes a plurality of second sub-objects. Each first sub-object and each second sub-object correspond to a corresponding sub-object pair to be processed. Each first data contained in the first sub-object and the corresponding second data contained in the second sub-object in the sub-object pair to be processed correspond to a data pair to be processed. Each sub-object pair to be processed is used to be assigned to a corresponding processing unit in the computing component for data processing.

[0241] Processing module 1302 is used to, for each data pair to be processed in the sub-object pair to be processed, in response to the first data or the second data in the data pair to be processed being invalid data, send indication information representing that the first data or the second data is invalid data to the processing unit corresponding to the sub-object pair to be processed, so that the processing unit skips data processing of the data pair to be processed based on the indication information.

[0242] In an optional embodiment, the processing unit corresponding to each pair of sub-objects to be processed is configured to perform data processing on each pair of data to be processed contained in the pair of sub-objects to be processed in a serial manner through a corresponding processing cycle; the processing unit processes one pair of data to be processed in one processing cycle;

[0243] Each first sub-object in the first data object corresponds to a first primary record and a first backup record, and each second sub-object in the second data object corresponds to a second primary record and a second backup record;

[0244] The first primary record can be used to record first data to be processed by the corresponding first sub-object in the current processing cycle, and the first backup record can be used to record first data corresponding to the corresponding first sub-object in the corresponding historical processing cycle; the second primary record can be used to record second data to be processed by the corresponding second sub-object in the current processing cycle, and the second backup record can be used to record second data corresponding to the corresponding second sub-object in the corresponding historical processing cycle;

[0245] The processing module 1302 is further configured to:

[0246] In response to the switching processing cycle, data update processing is performed on the corresponding register corresponding to the first sub-object, and data update processing is performed on the corresponding register corresponding to the second sub-object, and based on the data in each updated register, the first data and the second data are distributed to the corresponding processing unit to form a pair of data to be processed in the corresponding processing unit and perform data processing on the pair of data to be processed.

[0247] In an optional embodiment, when the processing module 1302 performs data updating processing on the corresponding register corresponding to the first sub-object, performs data updating processing on the corresponding register corresponding to the second sub-object, and distributes the first data and the second data to the corresponding processing unit based on the data in each updated register, it is specifically configured to:

[0248] In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being invalid data, not updating the data of the main register and the backup register corresponding to the corresponding sub-object, not allocating the current data to be processed to the corresponding processing unit in the current processing cycle, and generating indication information indicating that the current data to be processed is invalid data, and issuing the indication information to the corresponding processing unit;

[0249] In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being valid data, updating the main register corresponding to the corresponding sub-object based on at least the current data to be processed, and distributing the data in the main register and the backup register corresponding to the corresponding sub-object to the corresponding processing unit, so that the processing unit selects required data from the data in the main register and the backup register corresponding to the corresponding sub-object;

[0250] The corresponding sub-object is the corresponding first sub-object or the second sub-object.

[0251] In an optional implementation manner, when skipping data processing of the to-be-processed data pair based on the instruction information, the processing unit is configured to:

[0252] For the current processing cycle, the processing unit, based on the instruction information, does not update the first data branch corresponding to the invalid data, uses data allocated from the spare register of the corresponding sub-object to update the second data branch other than the first data branch corresponding to the invalid data, and avoids operating on the to-be-processed data pair corresponding to the invalid data;

[0253] Each processing unit corresponds to two data branches, which are respectively used to provide the processing unit with corresponding first data and second data in the processing cycle, so as to form the required data pair to be processed.

[0254] In an optional embodiment, when the processing unit uses the data allocated from the spare register of the corresponding sub-object to perform data update processing on the second data branch other than the first data branch corresponding to the invalid data, it is specifically configured to:

[0255] In response to the data allocated from the spare register of the corresponding sub-object being identical to the current data in the second data branch, not updating the data in the second data branch;

[0256] In response to the data allocated from the spare register of the corresponding sub-object being different from the data in the second data branch, the data in the second data branch is updated to the data allocated from the spare register of the corresponding sub-object.

[0257] In an optional implementation manner, when the processing module 1302 updates the main register corresponding to the corresponding sub-object based at least on the current data to be processed, it is specifically configured to:

[0258] The data in the standby register corresponding to the corresponding sub-object is updated to the current data in the main register corresponding to the corresponding sub-object, and the data in the main register corresponding to the corresponding sub-object is updated to the current data to be processed.

[0259] In an optional implementation manner, when the processing module 1302 updates the main register corresponding to the corresponding sub-object based at least on the current data to be processed, it is specifically configured to:

[0260] determining whether the number of invalid data in each to-be-processed data corresponding to a current processing cycle of each sub-object in another data object other than the data object to which the corresponding sub-object belongs reaches a threshold, and if so, not updating the standby register corresponding to the corresponding sub-object; if not, updating the data in the standby register corresponding to the corresponding sub-object with the current data in the main register corresponding to the corresponding sub-object;

[0261] The data in the main register corresponding to the corresponding sub-object is updated to the current data to be processed.

[0262] The present application also discloses an electronic device, the composition structure of the electronic device is as follows: Figure 14 As shown, it includes the data processing chip 1401 described above, so as to execute the data processing method disclosed in any method embodiment of the present application based on the data processing chip 1401.

[0263] In addition, optionally, the electronic device may further include a memory 1402 and a processor 1403 .

[0264] The memory 1402 is used to store computer instruction sets and / or data.

[0265] A set of computer instructions can be implemented in the form of a computer program.

[0266] Processor 1403 is used to perform relevant data processing.

[0267] The processor 1403 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices.

[0268] Optionally, the electronic device also includes storage resources such as memory and cache.

[0269] Optionally, the electronic device further includes a camera assembly, and / or is connected to an external camera assembly.

[0270] In addition, the electronic device may also include components such as a communication interface and a communication bus. The memory, processor, and communication interface communicate with each other via the communication bus.

[0271] Communication interfaces are used for communication between electronic devices and other devices. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, and a control bus.

[0272] In addition, an embodiment of the present application further discloses a storage medium, which carries one or more computer instruction sets. When the one or more computer instruction sets are executed by an electronic device, the electronic device can implement a data processing method as provided in any of the above method embodiments.

[0273] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.

[0274] For the convenience of description, the above systems or devices are described as being divided into various modules or units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0275] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0276] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further limitations, an element limited by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0277] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing chip, comprising: A computing component, including an array of processing units; Controller for: Obtain a first data object and a second data object; The first data object includes a plurality of first sub-objects, and the second data object includes a plurality of second sub-objects. Each first sub-object and each second sub-object correspond to a corresponding sub-object pair to be processed. Each first data contained in the first sub-object and the corresponding second data contained in the second sub-object in the sub-object pair to be processed correspond to a data pair to be processed. Each sub-object pair to be processed is used to be assigned to a corresponding processing unit in the computing component for data processing. For each data pair to be processed in the sub-object pair to be processed, in response to the first data or the second data in the data pair to be processed being invalid data, indication information representing that the first data or the second data is invalid data is sent to the processing unit corresponding to the sub-object pair to be processed, so that the processing unit skips data processing of the data pair to be processed based on the indication information.

2. The data processing chip according to claim 1, wherein the processing unit corresponding to each sub-object pair to be processed is configured to perform data processing on each data pair to be processed contained in the sub-object pair to be processed in a serial manner through a corresponding processing cycle; the processing unit processes one data pair to be processed in one processing cycle; The data processing chip further includes a first main register unit group, a first backup register unit group, a second main register unit group, and a second backup register unit group; each processing unit group in the first direction of the processing unit array corresponds to a first main register unit and a first backup register unit, for processing a first sub-object in the first data object; each processing unit group in the second direction of the processing unit array corresponds to a second main register unit and a second backup register unit, for processing a second sub-object in the second data object; The first main register unit can be used to store the first data to be processed by the corresponding first sub-object in the current processing cycle, and the first backup register unit can be used to store the first data corresponding to the corresponding first sub-object in the corresponding historical processing cycle; the second main register unit can be used to store the second data to be processed by the corresponding second sub-object in the current processing cycle, and the second backup register unit can be used to store the second data corresponding to the corresponding second sub-object in the corresponding historical processing cycle.

3. The data processing chip according to claim 2, wherein the controller is further configured to: In response to the switching processing cycle, data update processing is performed on the corresponding storage unit corresponding to the first sub-object, and data update processing is performed on the corresponding storage unit corresponding to the second sub-object, and based on the data in each storage unit after the update, the first data and the second data are distributed to the corresponding processing unit to form a pair of data to be processed in the corresponding processing unit and perform data processing on the pair of data to be processed.

4. The data processing chip according to claim 3, wherein the controller, when performing data updating processing on the corresponding register unit corresponding to the first sub-object, performing data updating processing on the corresponding register unit corresponding to the second sub-object, and distributing the first data and the second data to the corresponding processing unit based on the updated data in each register unit, is configured to: In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being invalid data, not updating data in the primary register unit and the backup register unit corresponding to the corresponding sub-object, not allocating the current data to be processed to the corresponding processing unit in the current processing cycle, generating indication information indicating that the current data to be processed is invalid data, and issuing the indication information to the corresponding processing unit; In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being valid data, updating the primary register unit corresponding to the corresponding sub-object based on at least the current data to be processed, and distributing the data in the primary register unit and the backup register unit corresponding to the corresponding sub-object to the corresponding processing unit, so that the processing unit selects required data from the data in the primary register unit and the backup register unit corresponding to the corresponding sub-object; in, The corresponding sub-object is the corresponding first sub-object or the second sub-object.

5. The data processing chip according to claim 4, wherein the processing unit comprises: a first receiving unit, configured to receive first data allocated by the controller based on the corresponding first active register unit and the first standby register unit, and second indication information indicating that second data to be processed in a current processing cycle is invalid data; In response to not receiving the first data, not updating the data on the first data branch; and in response to receiving the first data and the second indication information, updating the data on the first data branch based on the first data in the first spare register unit; In response to receiving the first data and not receiving the second indication information, updating the data in the first data branch based on the first data in the first main register unit; a second receiving unit, configured to receive second data allocated by the controller based on the corresponding second active register unit and the second standby register unit, and first indication information indicating that first data to be processed in a current processing cycle is invalid data; In response to not receiving the second data, not updating the data on the second data branch; in response to receiving the second data and the first indication information, updating the data on the second data branch based on the second data in the second standby register unit; and in response to receiving the second data and not receiving the first indication information, updating the data in the second data branch based on the second data in the second active register unit; a first operation unit, configured to, in response to data update occurring in the first data branch and the second data branch, perform a multiplication operation on a data pair formed by the data in the first data branch and the second data branch, and, in response to no data update occurring in the first data branch and / or the second data branch, not perform the multiplication operation on the data in the first data branch and the second data branch; a second operation unit, configured to accumulate the multiplication result of the first operation unit and the first multiplication-accumulation result stored in the register unit to obtain a second multiplication-accumulation result and output the result; A storage unit is used to store the output data of the second operation unit.

6. The data processing chip according to claim 5, wherein the processing unit further comprises: a control unit, configured to receive the first indication information and the second indication information, generate a target gating signal in response to receiving the first indication information and / or the second indication information, and send the target gating signal to the register unit; The target gating signal is used to instruct the register unit not to update data in the current processing cycle.

7. The data processing chip according to claim 1, wherein the first data object and the second data object are matrices, each first sub-object is a row in the first data object, and each second sub-object is a column in the second data object; The same first data in each row of the first data object is simultaneously distributed to the respective processing units in the same row of the processing unit array in a corresponding processing cycle, and each processing unit in the same row of the processing unit array is used to process the respective first data in the same row of the first data object; The same second data in each column of the second data object is simultaneously distributed to the respective processing units in the same column of the processing unit array in a corresponding processing cycle, and each processing unit in the same column of the processing unit array is used to process the respective second data in the same column of the second data object; Each processing unit in the processing unit array forms a pair of data to be processed based on the corresponding first data and second data obtained, and each processing unit in the processing unit array processes its own pair of data to be processed in parallel in the same processing cycle; The controller is configured to, when sending indication information indicating that the first data or the second data is invalid data to the processing unit corresponding to the sub-object to be processed: Based on the broadcast mode, indication information indicating that the first data or the second data is invalid data is sent to each processing unit in the data unit array that uses the first data or the second data as one of the input data.

8. The data processing chip according to claim 4, wherein when the controller updates the primary register unit corresponding to the corresponding sub-object based at least on the current data to be processed, the controller is configured to: The data in the standby register unit corresponding to the corresponding sub-object is updated to the current data in the main register unit corresponding to the corresponding sub-object, and the data in the main register unit corresponding to the corresponding sub-object is updated to the current data to be processed.

9. The data processing chip according to claim 4, wherein when the controller updates the primary register unit corresponding to the corresponding sub-object based at least on the current data to be processed, the controller is configured to: determining whether the number of invalid data in each to-be-processed data corresponding to the current processing cycle of each sub-object in another data object other than the data object to which the corresponding sub-object belongs reaches a threshold; if so, not updating the standby register unit corresponding to the corresponding sub-object; if not, updating the data in the standby register unit corresponding to the corresponding sub-object with the current data in the main register unit corresponding to the corresponding sub-object; The data in the main register unit corresponding to the corresponding sub-object is updated to the current data to be processed.

10. A data processing method comprising: Obtain a first data object and a second data object; wherein the first data object includes a plurality of first sub-objects, the second data object includes a plurality of second sub-objects, each first sub-object and each second sub-object correspond to a corresponding pair of sub-objects to be processed, and each first data contained in the first sub-object and the corresponding second data contained in the second sub-object in the pair of sub-objects to be processed correspond to a pair of data to be processed; each pair of sub-objects to be processed is used to be assigned to a corresponding processing unit in the computing component for data processing, and the computing component includes a processing unit array formed by a plurality of processing units; For each data pair to be processed in the sub-object pair to be processed, in response to the first data or the second data in the data pair to be processed being invalid data, indication information representing that the first data or the second data is invalid data is sent to the processing unit corresponding to the sub-object pair to be processed, so that the processing unit skips data processing of the data pair to be processed based on the indication information.

11. The data processing method according to claim 10, wherein the processing unit corresponding to each pair of sub-objects to be processed is configured to perform data processing on each pair of data to be processed contained in the pair of sub-objects to be processed in a serial manner through a corresponding processing cycle; The processing unit processes one pair of data to be processed in one processing cycle; Each first sub-object in the first data object corresponds to a first primary record and a first backup record, and each second sub-object in the second data object corresponds to a second primary record and a second backup record; The first primary record can be used to record first data to be processed by the corresponding first sub-object in the current processing cycle, and the first backup record can be used to record first data corresponding to the corresponding first sub-object in the corresponding historical processing cycle; the second primary record can be used to record second data to be processed by the corresponding second sub-object in the current processing cycle, and the second backup record can be used to record second data corresponding to the corresponding second sub-object in the corresponding historical processing cycle; The data processing method further includes: In response to the switching processing cycle, data update processing is performed on the corresponding register corresponding to the first sub-object, and data update processing is performed on the corresponding register corresponding to the second sub-object, and based on the data in each updated register, the first data and the second data are distributed to the corresponding processing unit to form a pair of data to be processed in the corresponding processing unit and perform data processing on the pair of data to be processed.

12. The data processing method according to claim 11, wherein the updating of data in the corresponding register corresponding to the first sub-object and the updating of data in the corresponding register corresponding to the second sub-object, and distributing the first data and the second data to the corresponding processing units based on the updated data in each register, comprises: In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being invalid data, not updating the data of the main register and the backup register corresponding to the corresponding sub-object, not allocating the current data to be processed to the corresponding processing unit in the current processing cycle, and generating indication information indicating that the current data to be processed is invalid data, and issuing the indication information to the corresponding processing unit; In response to the current processing cycle being switched to and the current data to be processed corresponding to the corresponding sub-object being valid data, updating the main register corresponding to the corresponding sub-object based on at least the current data to be processed, and distributing the data in the main register and the backup register corresponding to the corresponding sub-object to the corresponding processing unit, so that the processing unit selects required data from the data in the main register and the backup register corresponding to the corresponding sub-object; The corresponding sub-object is the corresponding first sub-object or the second sub-object.

13. The data processing method according to claim 12, wherein the processing unit skips data processing of the to-be-processed data pair based on the instruction information, comprising: For the current processing cycle, the processing unit, based on the instruction information, does not update the first data branch corresponding to the invalid data, uses data allocated from the spare register of the corresponding sub-object to update the second data branch other than the first data branch corresponding to the invalid data, and avoids operating on the to-be-processed data pair corresponding to the invalid data; Each processing unit corresponds to two data branches, which are respectively used to provide the processing unit with corresponding first data and second data in the processing cycle, so as to form the required data pair to be processed.

14. The data processing method according to claim 13, wherein the step of updating the second data branch other than the first data branch corresponding to the invalid data using the data allocated from the spare register of the corresponding sub-object comprises: In response to the data allocated from the spare register of the corresponding sub-object being identical to the current data in the second data branch, not updating the data in the second data branch; In response to the data allocated from the spare register of the corresponding sub-object being different from the data in the second data branch, the data in the second data branch is updated to the data allocated from the spare register of the corresponding sub-object.

15. The data processing method according to claim 12, wherein updating the main register corresponding to the corresponding sub-object based at least on the current data to be processed comprises: The data in the standby register corresponding to the corresponding sub-object is updated to the current data in the main register corresponding to the corresponding sub-object, and the data in the main register corresponding to the corresponding sub-object is updated to the current data to be processed.

16. The data processing method according to claim 12, wherein updating the main register corresponding to the corresponding sub-object based on at least the current data to be processed comprises: determining whether the number of invalid data in each to-be-processed data corresponding to a current processing cycle of each sub-object in another data object other than the data object to which the corresponding sub-object belongs reaches a threshold, and if so, not updating the standby register corresponding to the corresponding sub-object; if not, updating the data in the standby register corresponding to the corresponding sub-object with the current data in the main register corresponding to the corresponding sub-object; The data in the main register corresponding to the corresponding sub-object is updated to the current data to be processed.

17. A data processing device comprising: An acquisition module, configured to obtain a first data object and a second data object; The first data object includes a plurality of first sub-objects, and the second data object includes a plurality of second sub-objects. Each first sub-object and each second sub-object correspond to a corresponding sub-object pair to be processed. Each first data contained in the first sub-object and the corresponding second data contained in the second sub-object in the sub-object pair to be processed correspond to a data pair to be processed. Each sub-object pair to be processed is used to be assigned to a corresponding processing unit in the computing component for data processing. A processing module is used to, for each data pair to be processed in the sub-object pair to be processed, in response to the first data or the second data in the data pair to be processed being invalid data, send indication information representing that the first data or the second data is invalid data to the processing unit corresponding to the sub-object pair to be processed, so that the processing unit skips data processing of the data pair to be processed based on the indication information.

Images