Data processing method, device, storage medium and electronic device

Abstract

The embodiments of the present application disclose a data processing method, device, storage medium, and electronic device, wherein the embodiments of the present application determine a first weight vector to be calculated from the weight data of a target convolutional layer based on the capacity information of a multiplication-addition array; obtain a first zero-value mark sequence of the first weight vector, read non-zero-value weights from the first weight vector according to the first zero-value mark sequence, and assign the non-zero-value weights to the multiplication-addition array; read eigenvalues ​​matching the non-zero-value weights from the input feature data of the target convolutional layer according to the first zero-value mark sequence, and assign the eigenvalues ​​to the multiplication-addition array for calculation. By adopting the solution of the embodiments of the present application, the number of multiplication units involved in the calculation of the multiplication-addition array in one operation is reduced, saving computing resources and data transmission bandwidth.

Description

Technical Field

[0001] The present application relates to the technical field of electronic equipment, and in particular to a data processing method, device, storage medium and electronic equipment. Background Art

[0002] The core operation in AI networks is the convolution operation between tensors, which involves numerous multiplication-accumulation operations, typically between the input feature map and the convolution kernel. In actual operations, the convolution kernel weights often contain a certain number of zero values, wasting transmission bandwidth and a large number of computing units. Summary of the Invention

[0003] The embodiments of the present application provide a data processing method, device, storage medium, and electronic device, which can save computing units and transmission bandwidth.

[0004] In a first aspect, an embodiment of the present application provides a data processing method, comprising:

[0005] Determine a first weight vector to be calculated from the weight data of the target convolutional layer according to the capacity information of the multiply-add array;

[0006] Obtaining a first zero-valued marker sequence of the first weight vector, reading non-zero-valued weights from the first weight vector according to the first zero-valued marker sequence, and assigning the non-zero-valued weights to the multiply-add array;

[0007] According to the first zero-valued tag sequence, feature values ​​matching the non-zero-valued weights are read from the input feature data of the target convolutional layer, and the feature values ​​are assigned to the multiplication-addition array for operation.

[0008] In a second aspect, an embodiment of the present application further provides a data processing device, including:

[0009] a determination module, configured to determine a first weight vector to be calculated from weight data of a target convolutional layer according to capacity information of the multiplication-addition array;

[0010] a first scheduling module, configured to obtain a first zero-valued mark sequence of the first weight vector, read non-zero-valued weights from the first weight vector according to the first zero-valued mark sequence, and distribute the non-zero-valued weights to the multiplication-addition array;

[0011] A second scheduling module is used to read feature values ​​that match the non-zero weights from the input feature data of the target convolutional layer according to the first zero-value mark sequence, and assign the feature values ​​to the multiplication-addition array for operation.

[0012] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is run on a computer, the computer executes a data processing method as provided in any embodiment of the present application.

[0013] In a fourth aspect, an embodiment of the present application further provides an electronic device comprising a processor and a memory, wherein the memory has a computer program, and the processor is configured to execute a data processing method as provided in any embodiment of the present application by calling the computer program.

[0014] The technical solution provided by the embodiment of the present application is as follows: when calculating the target convolution layer, a first weight vector is determined from the weight data according to the capacity information of the multiplication-addition array, and a corresponding first zero-value mark sequence is obtained at the same time. Non-zero-value weights are read from the first weight vector according to the sequence, and the non-zero-value weights are assigned to the multiplication-addition array. Then, eigenvalues ​​corresponding to the non-zero-value weights are read from the input feature data according to the sequence, and the read eigenvalues ​​are correspondingly assigned to the multiplication-addition array. In this way, for the zero-value weights and their corresponding eigenvalues, there is no need to read them from the memory into the multiplication-addition array for calculation, which reduces the number of multiplication units involved in the calculation of the multiplication-addition array in one operation, saving computing resources and data transmission bandwidth. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A first flow chart of the data processing method provided in an embodiment of the present application.

[0017] Figure 2 A schematic diagram of the hardware structure of the data processing method provided in an embodiment of the present application.

[0018] Figure 3 A schematic diagram of a convolution operation in the data processing method provided in an embodiment of the present application.

[0019] Figure 4 A schematic diagram of another convolution operation in the data processing method provided in an embodiment of the present application.

[0020] Figure 5 A schematic diagram of reading weight values ​​and eigenvalues ​​in the data processing method provided in an embodiment of the present application.

[0021] Figure 6A schematic diagram of integrating multiple zero-value distributions in the data processing method provided in an embodiment of the present application.

[0022] Figure 7 This is the traversal order of feature vectors in the data processing method provided in the embodiment of the present application.

[0023] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application.

[0024] Figure 9 This is a schematic diagram of the first structure of the electronic device provided in an embodiment of the present application.

[0025] Figure 10 A second structural diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0027] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] The embodiment of the present application provides a data processing method, and the execution subject of the data processing method can be the data processing device provided in the embodiment of the present application, or an electronic device integrated with the data processing device, wherein the data processing device can be implemented in hardware or software. Among them, the electronic device can be an intelligent terminal integrated with a convolutional neural network computing chip, such as a smart phone, tablet computer, PDA, laptop computer, desktop computer, intelligent vehicle-mounted device, intelligent monitoring equipment, etc. Alternatively, the electronic device can also be a server, and the user uploads the trained convolutional neural network to the server, and the server can use the trained convolutional neural network to perform calculations based on the data processing method of the present application.

[0029] See also Figure 1 , Figure 1 This is a schematic diagram of the first flow chart of the data processing method provided in the embodiment of the present application. The specific flow chart of the data processing method provided in the embodiment of the present application can be as follows:

[0030] 101. Determine a first weight vector to be calculated from weight data of a target convolutional layer according to capacity information of the multiplication-addition array.

[0031] The embodiments of the present application can be applied to the calculation process of a convolutional neural network of any structure. The convolutional neural network has been trained and can be used to process images, for example, the recognition of specific objects in an image, or image classification and other scenarios. The convolutional neural network includes at least one convolutional layer. For example, the embodiments of the present application can be applied to the calculation of a convolutional neural network with only one convolutional layer, and can also be applied to some complex convolutional neural networks, such as a convolutional neural network including up to hundreds or more convolutional layers. In addition, the convolutional neural network in the embodiments of the present application can also have a pooling layer, a fully connected layer, etc. In other words, the data processing method in the embodiments of the present application is not limited to a specific convolutional neural network. As long as it is a neural network containing a convolutional layer, it can be considered as a convolutional neural network in the present application, and its convolutional layer part can be calculated according to the data processing method of the embodiments of the present application.

[0032] A convolutional layer typically includes one or more filters, each consisting of multiple convolution kernels stacked along the channel direction. The weight data of the convolutional layer is the weight value in the filter. After training, the weight values ​​in the network are fixed, and some weights may be zero. Alternatively, in some cases, to increase the computational speed of the convolutional neural network, the convolutional neural network is compressed, and some weight values ​​in the compressed filter are converted to zero. The higher the compression level, the more zero values ​​there are.

[0033] During the convolution operation, when the zero weight value is multiplied by the eigenvalue in the input feature data, no matter what the eigenvalue is, the result is equal to zero, which not only does not contribute to the convolution result, but also wastes the performance and computing resources of the hardware. In general, the computing power provided by the electronic device equipped with the convolutional neural network is limited. For example, if the computing power of the multiplication and addition array of the electronic device is 256, then the multiplication and addition array has 256 multiplication units, that is, only 256 weight values ​​can be multiplied with the corresponding 256 eigenvalues ​​at the same time. Assuming that 100 of the 256 weight values ​​input to the multiplication and addition array are all zero, then the resources of 100 multiplication units will be wasted, because the result after the multiplication operation is zero, which has no effect on the subsequent multiplication and accumulation. When there are more zero weight values ​​in the entire convolutional neural network, the effective utilization rate of the multiplication and addition array is extremely low, which in turn causes the low computing efficiency of the entire convolutional neural network. Furthermore, suppose that 100 of the 256 weight values ​​input to the multiply-add array are all zero. These 100 zero values ​​and their corresponding eigenvalues ​​have no effect in the subsequent multiplication and accumulation, but they still occupy the data transmission bandwidth between the memory and the multiply-add array, and also cause a waste of transmission bandwidth.

[0034] The solution proposed in the embodiment of this application solves the above-mentioned problem, and the specific principles are as follows.

[0035] First, the hardware structure in the embodiment of the present application is described. Figure 2 As shown, Figure 2 A schematic diagram of the hardware structure of the data processing method provided in the embodiment of the present application. The basic hardware architecture of the embodiment of the present application includes a weight data storage module, a feature map storage module, an intermediate value storage module, a scheduling module and a multiplication-addition array. A multiplication-addition array may include multiple multiplication units. In this embodiment, in order to facilitate the reader's understanding of the solution, the multiplication-addition array is taken as an example including 8×8 multiplication units. Among them, the weight data storage module stores the weight data of the target convolution layer, the feature map storage module stores the input feature map data of the target convolution layer, the intermediate value storage module is used to store the intermediate accumulation result of the multiplication-addition column, and the scheduling module is used to control the data flow during the entire operation process.

[0036] like Figure 2 As shown, from K0C0 to K7C7, the multiplication-addition array in the figure includes 64 multiplication units. In one operation of the multiplication-addition array, one multiplication unit is responsible for the multiplication operation between a weight value and its corresponding eigenvalue. Since the multiplication-addition array can only perform 8 accumulations on the input channel at a time, for a vertical column of multiplication units, the accumulation result can be stored in the intermediate value storage module, and the final result can be output after the accumulation of the results of all 64 channels is completed.

[0037] The target convolution layer in the embodiment of the present application refers to the convolution layer currently performing calculations. When the target convolution layer is performing calculations, for a filter in the convolution layer, the filter is convolved on the input feature map of the target convolution layer to obtain a feature map of a channel on the output feature map. If there are N filters, the number of channels of the output feature map is N.

[0038] In some embodiments, the target convolution layer includes multiple filters; according to the capacity information of the multiplication-addition array, the step of determining the first weight vector to be calculated from the weight data of the target convolution layer may include: according to the capacity information of the multiplication-addition array, determining n first weight vectors from the weight data of the target convolution layer, the length of the first weight vector is s, the multiplication-addition array includes s columns × n rows of multiplication units, the target convolution layer includes multiple filters, one first weight vector corresponds to one filter, and the elements of the first weight vector are distributed along the channel direction on the filter.

[0039] like Figure 3 As shown, Figure 3 A schematic diagram of a convolution operation in a data processing method provided in an embodiment of the present application. Assume that the number of channels of the input feature map of the target convolution layer is S, the number of channels of each filter of the target convolution layer is also S, and the target convolution layer has N filters. For N filters, the scheduler determines a weight vector of length s in the n channel directions from the weight data storage module, where one filter determines a weight vector. Among them, s and n are size parameters of the multiply-add array, s represents the amount of data that can be processed at one time on the input channel in the multiply-add array, and n represents the number of filters that can be processed at one time in the multiply-add array. Figure 2 For the multiply-add array shown, s=n=8.

[0040] For the convenience of explaining the data processing process of the embodiment of the present application, it is assumed that S=64, N=8, s=n=8. In practical applications, S, N, s and n can also be other values. The convolution process of each filter and the feature map is as follows Figure 4 As shown, Figure 4 A schematic diagram of another convolution operation in the data processing method provided in an embodiment of the present application shows that the eight blocks on the left are the 64 eigenvalues ​​along the channel direction of the input feature map, which are divided into 8 eigenvectors. The eight blocks on the right are the 64 weight values ​​along the channel direction of a filter, which are divided into 8 weight vectors, of which one rectangular block is a 1*1*8 vector. In one operation of the multiplication-addition array, a weight vector of length 8 in the channel direction is determined from each of the 8 filters to obtain 8 weight vectors. These 8 weight vectors are the weight vectors to be operated and are mapped to a column of multiplication units in the multiplication-addition array.

[0041] For example, consider a filter size of 3×3×64 and a target convolutional layer with 8 filters. 3×3 represents length × width, and 64 represents the number of channels. This target convolutional layer can be divided into 3×3×8=72 first weight vectors. This means that the multiply-add array requires 72 operations to complete the target convolutional layer.

[0042] As can be seen from the above, the step of "determining the first weight vector to be calculated from the weight data of the target convolutional layer" means that for the current operation of the multiply-add array, the weight vectors to be calculated in this multiply-add array are determined from the target convolutional layer, without the need to read the specific weight values ​​from the memory. The operation of reading the weight values ​​is performed in a subsequent step.

[0043] 102. Obtain a first zero-valued marker sequence of the first weight vector, read non-zero-valued weights from the first weight vector according to the first zero-valued marker sequence, and distribute the non-zero-valued weights to a multiplication-addition array.

[0044] It should be noted that, since the values ​​of the various weights of the convolutional neural network are already determined before the operation, the convolutional neural network can be analyzed in advance, and the position information of each zero value in the filter can be recorded to generate a zero value mark. In addition, since the weight vector in the channel direction is generally read for operation during the convolution operation, in some embodiments, when recording the zero value label, the weight vector can be recorded as a unit, and the first zero value identification sequence of each first weight vector can be generated in advance. Wherein, the length of the first weight vector is equal to s, and the length of the first zero value mark sequence is also s. For example, the form of the first zero value mark sequence can be that when a certain weight value in the first weight vector is 0, the value at the corresponding position in the first zero value mark sequence is set to 1, and conversely, when the weight is not 0, the value of the UK position in the first zero value mark sequence is set to 0. For example, assuming that a certain first weight vector is (0.97, 1.23, 9.1, 0, 0, 4.5, 2.9, 3); then the corresponding first zero value mark sequence is (0, 0, 0, 1, 1, 0, 0, 0). That is to say, for a first weight vector, as long as the values ​​of the weights in the vector are known, the first zero-value mark sequence corresponding to the vector can be determined.

[0045] For the 8 first weight vectors determined above, the first zero value mark sequence corresponding to each first weight vector is obtained respectively. Then, for each first weight vector, according to the first zero value mark sequence corresponding to the first weight vector, the non-zero value weight is read from the first weight vector, and the read weight is assigned to the multiplication-addition array. Among them, a column (or a row) in the multiplication-addition array corresponds to a first weight array. For example, if 4 of the 8 weight values ​​of a first weight vector are zero values, it is only necessary to read the other 4 non-zero value weights from the weight data storage module and assign them to the four multiplication units of the corresponding columns of the first weight vector. In this way, the non-zero value weights in the 8 first weight vectors are read respectively and assigned to their corresponding columns in the multiplication-addition array.

[0046] 103. Read eigenvalues ​​matching the non-zero weights from the input feature data of the target convolutional layer according to the first zero-valued label sequence, and assign the eigenvalues ​​to a multiplication-addition array for operation.

[0047] Next, according to the first zero-valued tag sequence, the feature values ​​matching the non-zero weights are read from the input feature data of the target convolutional layer, such as Figure 5 As shown, Figure 5 The left side represents weight data, and the right side represents feature data. A first weight vector contains four non-zero values. This means the scheduling module reads four non-zero weights from the storage module and assigns them to the corresponding columns in the multiply-add array. Then, following the first zero-value marker sequence, it reads the four eigenvalues ​​at the same positions as shown in the diagram on the left from the feature vector that matches the first weight vector. These eigenvalues ​​are assigned to the four multiplication units where the four non-zero weights are located for computation.

[0048] For a target convolution layer, each filter in the target convolution layer needs to be convolved with the input feature map. The 8 columns of weight data input at a time in the multiplication-addition array come from different filters. Therefore, for one operation of the multiplication-addition array, after the 8 columns of weight data are assigned to the 8 columns of multiplication units, these 8 columns of weight data need to be operated with the same corresponding feature vector. For example, the 8 first weight vectors are the first weight vectors in the first row and first column of the 8 filters, then these 8 first weight vectors need to be calculated with the first feature vector in the first row and first column of the feature map.

[0049] For these 8 first weight vectors, when their zero value distributions are different, after the non-zero values ​​in the 8 first weight vectors are assigned to the multiplication-addition array, for the same eigenvector, it is necessary to read 8 times according to 8 different first zero value mark sequences, and obtain the eigenvalues ​​matching the non-zero value weights of each column, and assign them to the corresponding columns in the multiplication-addition array for calculation.

[0050] In order to improve the computing speed, in some embodiments, the steps of obtaining the first zero-value mark sequence of the first weight vector, reading the non-zero-value weight from the first weight vector according to the first zero-value mark sequence, and distributing the non-zero-value weight to the multiplication-addition array may include: obtaining the first zero-value mark sequence of each first weight vector in the n first weight vectors to obtain n first zero-value mark sequences, wherein the length of the first zero-value mark sequence is s; combining the n first zero-value mark sequences to obtain a second zero-value mark sequence; reading weight data from the n first weight vectors according to the second zero-value mark sequence to obtain n second weight vectors, and correspondingly distributing the n second weight vectors to the n-column multiplication-addition units of the multiplication-addition array.

[0051] In order to improve the computational efficiency of the convolutional neural network, after obtaining the first zero-value mark sequence corresponding to each first weight vector, a comprehensive analysis is performed on these n first zero-value mark sequences to generate a second zero-value mark sequence. This second zero-value mark sequence is used to uniformly represent the zero value distribution in these n first weight vectors. When reading feature data, it is only necessary to read the feature vector once according to the second zero-value mark sequence, without having to read it eight times.

[0052] After obtaining the second zero-value mark sequence, for the n first weight vectors, weight data is read from the n first weight vectors according to the second zero-value mark sequence to obtain n second weight vectors, and the n second weight vectors are correspondingly assigned to the n-column multiplication and addition units of the multiplication and addition array. The weight values ​​in these n second weight vectors are all non-zero values. If there is a zero value in the first weight vector, the zero-value weight is removed after the above operation, and the length of the second weight vector will be less than s, which can save some multiplication units. In addition, when reading the weight data from the memory, it is not necessary to read this part of the data, which solves the transmission bandwidth.

[0053] Among them, the step of synthesizing n first zero-value mark sequences to obtain a second zero-value mark sequence can be implemented in multiple ways. For example, for n first zero-value mark sequences, if a weight greater than a preset ratio at the same position is zero, a zero-value mark is added to the position; if a weight not greater than a preset ratio at the same position is zero, a non-zero-value mark is added to the position; based on the mark information corresponding to s positions, a second zero-value mark sequence of length s is generated. Among them, the position here refers to the sequence number of the element in the vector, for example, the first element in the vector, the second element in the vector, etc. Assuming n = 8, s = 8, and the preset ratio is 0, 25, if the first elements of 3 of the 8 first weight vectors are all zero, the first mark in the second zero-value sequence mark is recorded as a zero-value mark, that is, the first elements of these 8 weight vectors are all marked as zero.

[0054] For another example, a first zero-valued mark sequence having the largest number of zero-valued marks is determined from n first zero-valued mark sequences, and the determined first zero-valued mark sequence is used as the second zero-valued mark sequence.

[0055] For another example, for n first zero-valued mark sequences, if at least one weight value at the same position is zero, a zero-valued mark is added to the position; if the weights at the same position are all non-zero, a non-zero-valued mark is added to the position; based on the mark information corresponding to s positions, a second zero-valued mark sequence with a length of s is generated. Figure 6 As shown, by integrating the zero value distribution of the eight first weight vectors, a zero value distribution is obtained.

[0056] In some embodiments, according to a first zero-value mark sequence, eigenvalues ​​matching non-zero value weights are read from the input feature data of the target convolution layer, and the eigenvalues ​​are assigned to a multiplication-addition array for operation, which may include: according to the operation progress of the target convolution layer and a second zero-value mark sequence, reading eigenvectors from the input feature data of the target convolution layer, and assigning the eigenvectors to n-column multiplication-addition units for operation.

[0057] When performing convolution operations, the filter moves on the feature map in a certain order and performs convolution operations, so that all feature data on the input feature map participate in the operation. Based on this principle, the feature vectors will also be traversed in a certain order, such as Figure 7 The figure shows the order in which the feature vectors are traversed. The current position of the filter on the target convolutional layer indicates the current computational progress of the target convolutional layer. Based on the computational progress of the target convolutional layer and the second zero-valued marker sequence, a feature vector is read from the input feature data of the target convolutional layer. The length of this feature vector is equal to the length of the corresponding first weight vector after removing the zero values. This feature vector is then assigned to the n-column multiplication-addition unit for computation.

[0058] It should be noted that if the first weight vector is processed to remove zero values ​​according to the same second zero value marker sequence, the feature vector only needs to be read once, and the same feature vector is allocated to the 8-column multiplication and addition units.

[0059] The above process describes how to read weight values ​​and eigenvalues ​​in one operation of the multiplication-addition array. After step 103, the method further includes:

[0060] The calculation result is stored in the intermediate value memory, and the step of determining the first weight vector to be calculated from the weight data of the target convolution layer according to the capacity information of the multiplication-addition array is returned to execute until the calculation of the target convolution layer is completed.

[0061] After one operation of the multiplication-addition array is completed, the accumulated result of each column in this multiplication-addition operation is stored in the intermediate value memory, and the process returns to step 101 to continue obtaining new weight data for operation until all operations of the target convolutional layer are completed and the corresponding output feature map is obtained.

[0062] During specific implementation, the present application is not limited by the execution order of the various steps described. If no conflict occurs, some steps can be performed in other orders or simultaneously.

[0063] From the above, it can be seen that the data processing method provided by the embodiment of the present application, when calculating the target convolution layer, determines the first weight vector from the weight data according to the capacity information of the multiplication-addition array, and obtains the corresponding first zero-value mark sequence at the same time, reads the non-zero-value weight from the first weight vector according to the sequence, assigns the non-zero-value weight to the multiplication-addition array, and then reads the eigenvalue corresponding to the non-zero-value weight from the input feature data according to the sequence, and then assigns the read eigenvalue to the multiplication-addition array accordingly. In this way, for the zero-value weight and its corresponding eigenvalue, it is not necessary to read it from the memory to the multiplication-addition array for calculation, which reduces the number of multiplication units participating in the calculation of the multiplication-addition array in one operation, saving computing resources and data transmission bandwidth.

[0064] In one embodiment, a data processing device is also provided. Figure 8 , Figure 8 This is a schematic diagram of the structure of a data processing device 300 provided in an embodiment of the present application. The data processing device 300 is applied to an electronic device and includes modules 301, 302, 303, and 304, as follows:

[0065] A determination module 301 is configured to determine a first weight vector to be calculated from weight data of a target convolutional layer according to capacity information of a multiplication-addition array;

[0066] A first scheduling module 302 is configured to obtain a first zero-valued marker sequence of the first weight vector, read non-zero-valued weights from the first weight vector according to the first zero-valued marker sequence, and assign the non-zero-valued weights to the multiplication-addition array;

[0067] The second scheduling module 303 is used to read the feature values ​​matching the non-zero weights from the input feature data of the target convolutional layer according to the first zero-value mark sequence, and assign the feature values ​​to the multiplication-addition array for operation.

[0068] In some embodiments, the determination module 301 is further used to: determine n first weight vectors from the weight data of the target convolutional layer according to the capacity information of the multiplication-addition array, the length of the first weight vector is s, the multiplication-addition array includes s columns × n rows of multiplication units, the target convolutional layer includes multiple filters, one first weight vector corresponds to one filter, and the elements of the first weight vector are distributed along the channel direction on the filter.

[0069] In some embodiments, the first scheduling module 302 is further used to: obtain a first zero-value mark sequence of each first weight vector in the n first weight vectors to obtain n first zero-value mark sequences, wherein the length of the first zero-value mark sequence is s; combine the n first zero-value mark sequences to obtain a second zero-value mark sequence; and read weight data from the n first weight vectors according to the second zero-value mark sequence to obtain n second weight vectors, and distribute the n second weight vectors accordingly to the n columns of multiplication-addition units of the multiplication-addition array.

[0070] In some embodiments, the first scheduling module 302 is further used to: read feature vectors from the input feature data of the target convolution layer according to the operation progress of the target convolution layer and the second zero-value mark sequence, and distribute the feature vectors to the n-column multiplication and addition units for operation.

[0071] In some embodiments, the first scheduling module 302 is also used to: for the n first zero-value mark sequences, if a weight greater than a preset proportion at the same position is zero, then add a zero-value mark to the position; if a weight not greater than a preset proportion at the same position is zero, then add a non-zero value mark to the position; and, based on the mark information corresponding to the s positions, generate a second zero-value mark sequence with a length of s.

[0072] In some embodiments, the first scheduling module 302 is further configured to: determine a first zero-valued mark sequence having the largest number of zero-valued marks from the n first zero-valued mark sequences, and use the determined first zero-valued mark sequence as the second zero-valued mark sequence.

[0073] In some embodiments, the apparatus further comprises:

[0074] A data storage module is used to store the calculation result in the intermediate value memory, and return to execute the step of determining the first weight vector to be calculated from the weight data of the target convolution layer according to the capacity information of the multiplication and addition array until the calculation of the target convolution layer is completed.

[0075] It should be noted that the data processing device provided in the embodiment of the present application and the data processing method in the above embodiment belong to the same concept. Any method provided in the data processing method embodiment can be implemented through the data processing device. The specific implementation process is detailed in the data processing method embodiment and will not be repeated here.

[0076] From the above, it can be seen that the data processing device proposed in the embodiment of the present application, when calculating the target convolution layer, determines the first weight vector from the weight data according to the capacity information of the multiplication-addition array, and obtains the corresponding first zero-value mark sequence at the same time, reads the non-zero-value weight from the first weight vector according to the sequence, assigns the non-zero-value weight to the multiplication-addition array, and then reads the eigenvalue corresponding to the non-zero-value weight from the input feature data according to the sequence, and then assigns the read eigenvalue to the multiplication-addition array accordingly. In this way, for the zero-value weight and its corresponding eigenvalue, it is not necessary to read it from the memory to the multiplication-addition array for calculation, which reduces the number of multiplication units involved in the calculation of the multiplication-addition array in one operation, saving computing resources and data transmission bandwidth.

[0077] The present application also provides an electronic device. The electronic device may be a smart phone, tablet computer, or other device. Figure 9 , Figure 9 This is a schematic diagram of a first structure of an electronic device provided in an embodiment of the present application. The electronic device 400 includes a processor 401 and a memory 402. The processor 401 is electrically connected to the memory 402.

[0078] The processor 401 is the control center of the electronic device 400. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or calling computer programs stored in the memory 402 and calling data stored in the memory 402, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole.

[0079] Memory 402 can be used to store computer programs and data. The computer programs stored in memory 402 contain instructions that can be executed by the processor. Computer programs can be composed of various functional modules. Processor 401 executes various functional applications and data processing by calling the computer programs stored in memory 402.

[0080] In this embodiment, the processor 401 in the electronic device 400 loads instructions corresponding to one or more computer program processes into the memory 402 according to the following steps, and the processor 401 runs the computer program stored in the memory 402 to implement various functions:

[0081] Determine a first weight vector to be calculated from the weight data of the target convolutional layer according to the capacity information of the multiply-add array;

[0082] Obtaining a first zero-valued marker sequence of the first weight vector, reading non-zero-valued weights from the first weight vector according to the first zero-valued marker sequence, and assigning the non-zero-valued weights to the multiply-add array;

[0083] According to the first zero-valued tag sequence, feature values ​​matching the non-zero-valued weights are read from the input feature data of the target convolutional layer, and the feature values ​​are assigned to the multiplication-addition array for operation.

[0084] In some embodiments, see Figure 10 , Figure 10 This is a schematic diagram of a second structure of an electronic device provided in an embodiment of the present application. The electronic device 400 further includes a radio frequency circuit 403, a display screen 404, a control circuit 405, an input unit 406, an audio circuit 407, a sensor 408, and a power supply 409. The processor 401 is electrically connected to the radio frequency circuit 403, the display screen 404, the control circuit 405, the input unit 406, the audio circuit 407, the sensor 408, and the power supply 409, respectively.

[0085] The radio frequency circuit 403 is used to transmit and receive radio frequency signals to communicate with network devices or other electronic devices through wireless communication.

[0086] The display screen 404 may be used to display information input by a user or information provided to a user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces may be composed of images, texts, icons, videos, and any combination thereof.

[0087] The control circuit 405 is electrically connected to the display screen 404 and is used to control the display screen 404 to display information.

[0088] The input unit 406 may be configured to receive input numbers, characters, or user characteristics (e.g., fingerprints), and generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. The input unit 406 may include a fingerprint recognition module.

[0089] The audio circuit 407 can provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 407 includes a microphone. The microphone is electrically connected to the processor 401. The microphone is used to receive voice information input by the user.

[0090] The sensor 408 is used to collect external environment information. The sensor 408 may include one or more sensors such as an ambient brightness sensor, an acceleration sensor, and a gyroscope.

[0091] The power supply 409 is used to supply power to various components of the electronic device 400. In some embodiments, the power supply 409 can be logically connected to the processor 401 through a power management system, so that the power management system can manage charging, discharging, and power consumption.

[0092] Although not shown in the figure, the electronic device 400 may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0093] In this embodiment, the processor 401 in the electronic device 400 loads instructions corresponding to one or more computer program processes into the memory 402 according to the following steps, and the processor 401 runs the computer program stored in the memory 402 to implement various functions:

[0094] Determine a first weight vector to be calculated from the weight data of the target convolutional layer according to the capacity information of the multiply-add array;

[0095] Obtaining a first zero-valued marker sequence of the first weight vector, reading non-zero-valued weights from the first weight vector according to the first zero-valued marker sequence, and assigning the non-zero-valued weights to the multiply-add array;

[0096] According to the first zero-valued tag sequence, feature values ​​matching the non-zero-valued weights are read from the input feature data of the target convolutional layer, and the feature values ​​are assigned to the multiplication-addition array for operation.

[0097] As can be seen from the above, an embodiment of the present application provides an electronic device, which, when calculating the target convolution layer, determines a first weight vector from the weight data according to the capacity information of the multiplication-addition array, and simultaneously obtains the corresponding first zero-value mark sequence, reads the non-zero-value weight from the first weight vector according to the sequence, assigns the non-zero-value weight to the multiplication-addition array, and then reads the eigenvalue corresponding to the non-zero-value weight from the input feature data according to the sequence, and then assigns the read eigenvalue to the multiplication-addition array accordingly. In this way, for the zero-value weight and its corresponding eigenvalue, it is not necessary to read it from the memory to the multiplication-addition array for calculation, which reduces the number of multiplication units involved in the calculation of the multiplication-addition array in one operation, saving computing resources and data transmission bandwidth.

[0098] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program runs on a computer, the computer executes the data processing method described in any of the above embodiments.

[0099] It should be noted that, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer-readable storage medium may include but is not limited to: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0100] Furthermore, the terms "first," "second," and "third," etc., in this application are used to distinguish between different objects, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or modules is not limited to the listed steps or modules, but rather some embodiments may include steps or modules not listed, or other steps or modules that are inherent to such process, method, product, or apparatus.

[0101] The above describes in detail the data processing method, device, storage medium, and electronic device provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core concept of the present application. At the same time, those skilled in the art may vary in the specific implementation methods and application scope based on the concept of the present application. In summary, the contents of this specification should not be construed as limiting the present application.

Claims

1. A data processing method, characterized in that: include: Determine a first weight vector to be calculated from the weight data of the target convolutional layer according to the capacity information of the multiply-add array; Obtaining a first zero-valued marker sequence of the first weight vector, reading non-zero-valued weights from the first weight vector according to the first zero-valued marker sequence, and assigning the non-zero-valued weights to the multiply-add array; Reading, according to the first zero-valued marker sequence, feature values ​​that match the non-zero-valued weights from the input feature data of the target convolutional layer, and assigning the feature values ​​to the multiply-add array for operation; The calculation result is stored in the intermediate value memory, and the step of determining the first weight vector to be calculated from the weight data of the target convolutional layer according to the capacity information of the multiplication-addition array is returned to execute until the calculation of the target convolutional layer is completed.

2. The method according to claim 1, wherein The target convolutional layer includes a plurality of filters; The step of determining a first weight vector to be calculated from weight data of a target convolutional layer according to capacity information of the multiplication-addition array includes: According to the capacity information of the multiplication-addition array, n first weight vectors are determined from the weight data of the target convolutional layer, where the length of the first weight vector is s, the multiplication-addition array includes s columns × n rows of multiplication units, the target convolutional layer includes multiple filters, one first weight vector corresponds to one filter, and the elements of the first weight vector are distributed along the channel direction on the filter.

3. The method according to claim 2, wherein The acquiring a first zero-valued mark sequence of the first weight vector, reading non-zero-valued weights from the first weight vector according to the first zero-valued mark sequence, and assigning the non-zero-valued weights to the multiply-add array includes: Obtaining a first zero-valued marker sequence of each of the n first weight vectors to obtain n first zero-valued marker sequences, wherein the length of the first zero-valued marker sequence is s; Combining the n first zero-valued mark sequences to obtain a second zero-valued mark sequence; Weight data is read from the n first weight vectors according to the second zero-value mark sequence to obtain n second weight vectors, and the n second weight vectors are correspondingly allocated to the n columns of multiply-add units of the multiply-add array.

4. The method according to claim 3, wherein The step of reading, according to the first zero-valued marker sequence, feature values ​​that match the non-zero-valued weights from the input feature data of the target convolutional layer, and assigning the feature values ​​to the multiplication-addition array for operation, includes: According to the operation progress of the target convolution layer and the second zero-value mark sequence, feature vectors are read from the input feature data of the target convolution layer, and the feature vectors are distributed to the n-column multiplication and addition units for operation.

5. The method according to claim 3, wherein The step of synthesizing the n first zero-valued mark sequences to obtain a second zero-valued mark sequence includes: For the n first zero-value mark sequences, if a weight greater than a preset proportion at the same position is zero, a zero-value mark is added to the position; if a weight not greater than a preset proportion at the same position is zero, a non-zero-value mark is added to the position; A second zero-valued marker sequence with a length of s is generated according to the marker information corresponding to the s positions.

6. The method according to claim 3, wherein The step of synthesizing the n first zero-valued mark sequences to obtain a second zero-valued mark sequence includes: A first zero-valued mark sequence with the largest number of zero-valued marks is determined from the n first zero-valued mark sequences, and the determined first zero-valued mark sequence is used as a second zero-valued mark sequence.

7. A data processing device, characterized in that: include: a determination module, configured to determine a first weight vector to be calculated from weight data of a target convolutional layer according to capacity information of the multiplication-addition array; a first scheduling module, configured to obtain a first zero-valued mark sequence of the first weight vector, read non-zero-valued weights from the first weight vector according to the first zero-valued mark sequence, and distribute the non-zero-valued weights to the multiplication-addition array; a second scheduling module, configured to read, from the input feature data of the target convolutional layer according to the first zero-valued label sequence, feature values ​​that match the non-zero-valued weights, and assign the feature values ​​to the multiplication-addition array for operation; A data storage module is used to store the calculation result in the intermediate value memory, and return to execute the step of determining the first weight vector to be calculated from the weight data of the target convolution layer according to the capacity information of the multiplication and addition array until the calculation of the target convolution layer is completed.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is run on a computer, the computer is caused to execute the data processing method according to any one of claims 1 to 6.

9. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, wherein: The processor is configured to execute the data processing method according to any one of claims 1 to 6 by calling the computer program.

Images