Data processing method and apparatus, terminal, and storage medium

Abstract

The present disclosure relates to a data processing method and device, a terminal and a storage medium. The method comprises: converting a first feature matrix output by a neural network based on N input data into a second feature matrix; wherein the first feature matrix comprises 1 row and N*M columns, and the second feature matrix comprises N rows and M columns; converting a first convolution matrix of the neural network into a second convolution matrix; wherein the first convolution matrix comprises 1 row and O*M columns, and the second convolution matrix comprises O rows and M columns; and performing convolution operation based on the second feature matrix and the second convolution matrix to obtain a third feature matrix. Through the method, the efficiency of the convolution operation is improved without increasing the memory.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to a data processing method and device, a terminal, and a storage medium. BACKGROUND

[0002] Neural networks have been widely applied, and with the processors of terminals such as mobile phones becoming more and more powerful, more and more manufacturers deploy convolutional neural networks to mobile terminals. This scenario has extremely high requirements for performance. It is well known that the parallel computing of a graphics processing unit (GPU) is very suitable for accelerating the operation of a neural network, and the GPU of a mobile terminal is no exception. Compared with a central processing unit (CPU), the GPU has higher performance advantages, so the GPU is generally used for mobile terminal computing acceleration.

[0003] The main computing amount (generally 80%) of a neural network is concentrated on convolution operation, so the optimization of convolution operation is crucial. SUMMARY

[0004] The present disclosure provides a data processing method and device, a terminal, and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a data processing method is provided, comprising:

[0006] converting a first feature matrix output by a neural network based on N input data into a second feature matrix; wherein the first feature matrix comprises 1 row and N*M columns; feature values of M channels of the nth input data are located in the (n-1)*M+1th column to the n*Mth column of the first feature matrix; n is a positive integer less than or equal to N, M is a positive integer, and the feature values of different channels refer to feature values obtained based on different feature extraction methods; the second feature matrix comprises N rows and M columns, wherein the M feature values of the nth input data are located in the (n-1)*N+1th row to the n*Nth row and the 1st to the Mth column;

[0007] converting a first convolution matrix of the neural network into a second convolution matrix; wherein the first convolution matrix comprises 1 row and O*M columns; M convolution kernels of an oth convolution layer of the neural network are located in the (o-1)*M+1th column to the o*Mth column of the first convolution matrix; o is a positive integer less than or equal to O; the second convolution matrix comprises O rows and M columns, wherein the M convolution kernels of the oth convolution layer are located in the (o-1)*O+1th row to the o*Oth row and the 1st to the Mth column;

[0008] performing a convolution operation based on the second feature matrix and the second convolution matrix to obtain a third feature matrix.

[0009] In some embodiments, a feature value in one channel of the first feature matrix carries H1*W1 feature data; a feature value in one channel of the second feature matrix carries H1*W1 feature data, and the second feature matrix includes (H1*N)*(W1*M) feature data.

[0010] The method further includes:

[0011] When the product of W1*M is not a multiple of P, 0 is inserted at the end of the W1*M columns of the second feature matrix to pad, so that the value of W1*M in the second feature matrix is a multiple of P, where P is the number of bytes involved in one operation.

[0012] In some embodiments, one of the convolution kernels in the first convolution matrix carries H2*W2 weight data, and the conversion of the first convolution matrix of the neural network into a second convolution matrix includes:

[0013] When W2 is not equal to 1, the convolution kernel in the first convolution matrix is converted into a (H2*W2)*1 convolution kernel, and the second convolution matrix carrying (H2*W2*O)*M weight data is obtained.

[0014] In some embodiments, the method further includes:

[0015] When the product of H2*W2*O is not a multiple of P, 0 is inserted at the end of the H2*W2*O rows of the second convolution matrix to pad, so that the value of (H2*W2*O) in the second convolution matrix is a multiple of P, where P is the number of bytes involved in one operation.

[0016] In some embodiments, the method further includes:

[0017] The feature values in the second feature matrix are grouped and stored; wherein the feature value grouping and storage includes:

[0018] The feature values of the M channels are equally divided into r equal parts to form M / r feature value combinations;

[0019] One feature value in the second feature matrix after grouping and storage carries H1*(W1*(M / r)) feature data, and the second feature matrix includes (H1*N)*(W1*(M / r)*r) feature data in total; wherein the channels of the feature values in one feature value combination are adjacent, and r is greater than 0.

[0020] Before the convolution operation is performed, the stored second feature matrix is read in the feature value combination as a reading unit.

[0021] In some embodiments, the method further comprises:

[0022] The convolution kernel grouping storage in the second convolution matrix; wherein the convolution kernel grouping storage comprises:

[0023] O / s equal parts of the O convolution layers are formed into O / s convolution kernel combinations;

[0024] One convolution kernel in the second convolution matrix after grouping storage carries ((O / s)*H2*W2)*1 weight data, and the second convolution matrix comprises ((O / s)*H2*W2*s)*M weight data in total; wherein the convolution layers to which the convolution kernels in one convolution kernel combination belong are adjacent in hierarchy, and s is greater than 0;

[0025] Before the convolution operation is performed, the stored second convolution matrix is read in the convolution kernel combination as a reading unit.

[0026] In some embodiments, the method further comprises:

[0027] The third feature matrix after the convolution is converted into a fourth feature matrix; wherein the third feature matrix comprises: N rows and Q columns, Q feature values after the convolution of the nth input data, located in the (n-1)*N+1 row to the n*N row, and the 1st to the Qth column, Q being a positive integer; and the fourth feature matrix comprises: 1 row and N*Q columns, Q feature values after the convolution of the nth input data, located in the (n-1)*Q+1 column to the n*Q column of the fourth feature matrix.

[0028] According to a second aspect of the embodiments of the present disclosure, a data processing apparatus is provided, comprising:

[0029] The first conversion module is configured to convert a first feature matrix output by a neural network based on N input data into a second feature matrix; wherein the first feature matrix comprises: 1 row and N*M columns; M feature values of the nth input data in the (n-1)*M+1 column to the n*M column of the first feature matrix; n is a positive integer less than or equal to N, M is a positive integer, and the feature values of different channels refer to feature values obtained based on different feature extraction methods; and the second feature matrix comprises: N rows and M columns, wherein M feature values of the nth input data are located in the (n-1)*N+1 row to the n*N row and the 1st to the Mth column.

[0030] a second conversion module, configured to convert the first convolution matrix of the neural network into a second convolution matrix; wherein the first convolution matrix comprises: 1 row and O*M columns; M convolution kernels of an oth convolution layer of the neural network, located in (o-1)*M+1th column to o*Mth column of the first convolution matrix; o is a positive integer less than or equal to O; the second convolution matrix comprises: O rows and M columns, wherein the M convolution kernels of the oth convolution layer are located in (o-1)*O+1th row to o*Oth row and 1st column to Mth column;

[0031] a convolution module, configured to perform convolution operation based on the second feature matrix and the second convolution matrix to obtain a third feature matrix.

[0032] In some embodiments, a feature value of one channel in the first feature matrix carries H1*W1 feature data; a feature value of one channel in the second feature matrix carries H1*W1 feature data, and the second feature matrix comprises (H1*N)*(W1*M) feature data, and the apparatus further comprises:

[0033] a first insertion module, configured to insert 0 at the end of W1*M columns of the second feature matrix to pad, when the product of W1*M is not a multiple of P, so that the value of W1*M in the second feature matrix is a multiple of P, wherein P is a number of bytes involved in one operation.

[0034] In some embodiments, one of the convolution kernels in the first convolution matrix carries H2*W2 weight data,

[0035] The second conversion module is specifically configured to convert the convolution kernel in the first convolution matrix into a (H2*W2)*1 convolution kernel when W2 is not equal to 1, to obtain the second convolution matrix carrying (H2*W2*O)*M weight data.

[0036] In some embodiments, the apparatus further comprises:

[0037] a second insertion module, configured to insert 0 at the end of H2*W2*O rows of the second convolution matrix to pad, when the product of H2*W2*O is not a multiple of P, so that the value of (H2*W2*O) in the second convolution matrix is a multiple of P, wherein P is a number of bytes involved in one operation.

[0038] In some embodiments, the apparatus further comprises:

[0039] The first storage module is configured to store feature value groups in the second feature matrix, wherein the feature value group storage comprises: equally dividing feature values of the M channels into r equal parts to form M / r feature value combinations; and one feature value in the second feature matrix after group storage carries H1*(W1*(M / r)) feature data, and the second feature matrix comprises (H1*N)*(W1*(M / r)*r) feature data in total; wherein the feature values in one feature value combination are adjacent in channel, and r is greater than 0.

[0040] The first reading module is configured to read the stored second feature matrix in the feature value combination as a reading unit before the convolution operation is performed.

[0041] In some embodiments, the apparatus further comprises:

[0042] The second storage module is configured to store convolution kernels in the second convolution matrix in groups; wherein the convolution kernel group storage comprises: equally dividing O convolution layers into s equal parts to form O / s convolution kernel combinations of the convolution layers; and one convolution kernel in the second convolution matrix after group storage carries ((O / s)*H2*W2)*1 weight data, and the second convolution matrix comprises ((O / s)*H2*W2*s)*M weight data in total; wherein the convolution layers to which the convolution kernels in one convolution kernel combination belong are adjacent in layer level, and s is greater than 0.

[0043] The second reading module is configured to read the stored second convolution matrix in the convolution kernel combination as a reading unit before the convolution operation is performed.

[0044] In some embodiments, the apparatus further comprises:

[0045] The third conversion module is configured to convert the third feature matrix after convolution into a fourth feature matrix; wherein the third feature matrix comprises: N rows and Q columns, Q feature values after convolution of the nth input data, located in the (n-1)*N+1 row to the n*N row, and the 1st to the Qth column, Q being a positive integer; and the fourth feature matrix comprises: 1 row and N*Q columns, Q feature values after convolution of the nth input data, located in the (n-1)*Q+1 column to the n*Q column of the fourth feature matrix.

[0046] According to a third aspect of the embodiments of the present disclosure, a terminal is provided, comprising:

[0047] a processor;

[0048] a memory for storing processor-executable instructions;

[0049] The processor is configured to perform the data processing method in the first aspect.

[0050] According to a fourth aspect of the embodiments of the present disclosure, a storage medium is provided, comprising:

[0051] When the instructions in the storage medium are executed by the processor of the terminal, the terminal can perform the data processing method in the first aspect.

[0052] The technical solutions provided by the embodiments of the present disclosure can include the following beneficial effects:

[0053] In the embodiments of the present disclosure, before performing convolution operation on data in a neural network, the terminal converts a first feature matrix including 1 row and N*M columns of feature values into a second feature matrix of N rows and M columns in two dimensions, wherein the feature values are taken as a whole as an element in the first feature matrix or the second feature matrix; similarly, a first convolution matrix including 1 row and O*M columns of convolution kernels is converted to obtain a second convolution matrix of O rows and M columns in two dimensions, wherein the convolution kernels are taken as a whole as an element in the first convolution matrix or the second convolution matrix. On the one hand, in the above conversion, only the rearrangement of the matrix is involved, without the need to increase additional memory to store intermediate results or temporary auxiliary variables, so that additional storage space is not increased; on the other hand, the second feature matrix and the second convolution matrix are converted by taking the feature values and the convolution kernels as elements, and the parallel operation in units of rows can be performed, so that the efficiency of the convolution operation can be improved.

[0054] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0055] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure.

[0056] Figure 1 is a data processing method according to an embodiment of the present disclosure.

[0057] Figure 2 is an example diagram of a convolution process for input data and parameter data in an embodiment of the present disclosure.

[0058] Figure 3 is an example diagram of a convolution operation method.

[0059] Figure 4 is an example diagram of an example format conversion of input data to be convolved in an embodiment of the present disclosure.

[0060] Figure 5 is an example diagram of a data processing method according to an example embodiment.

[0061] Figure 6 is an example diagram of a data processing method according to an example embodiment.

[0062] Figure 7 is a diagram of a data processing apparatus according to an example embodiment.

[0063] Figure 8 is a block diagram of a terminal apparatus according to an example embodiment. DETAILED DESCRIPTION

[0064] The example embodiments will be described in detail below with reference to the accompanying drawings. In the following description, the same numbers are used to denote the same elements, unless otherwise indicated. The implementations described in the following example embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0065] Figure 1 is an example diagram of a data processing method according to an example embodiment. Figure 1 As shown, the data processing method applied in the terminal comprises the following steps:

[0066] S11, converting a first feature matrix output by a neural network based on N input data into a second feature matrix; wherein the first feature matrix comprises 1 row and N*M columns; feature values of M channels of the nth input data are located in the (n-1)*M+1th column to the nth*Mth column of the first feature matrix; n is a positive integer less than or equal to N, M is a positive integer, and the feature values of different channels refer to feature values obtained based on different feature extraction methods; the second feature matrix comprises N rows and M columns, wherein the M feature values of the nth input data are located in the (n-1)*N+1th row to the nth*Nth row and the 1st to the Mth column;

[0067] S12, converting a first convolution matrix of the neural network into a second convolution matrix; wherein the first convolution matrix comprises 1 row and O*M columns; M convolution kernels of the oth convolution layer of the neural network are located in the (o-1)*M+1th column to the o*Mth column of the first convolution matrix; o is a positive integer less than or equal to O; the second convolution matrix comprises O rows and M columns, wherein the M convolution kernels of the oth convolution layer are located in the (o-1)*O+1th row to the o*Oth row and the 1st to the Mth column;

[0068] S13, performing a convolution operation based on the second feature matrix and the second convolution matrix to obtain a third feature matrix.

[0069] In embodiments of the present disclosure, the terminal device includes a mobile device and a fixed device; the mobile device includes a mobile phone, a tablet computer, a wearable device, and the like. The fixed device includes, but is not limited to, a personal computer (PC).

[0070] In step S11 of the present disclosure, the first feature matrix output by N input data is converted into a second feature matrix before the convolution operation of the neural network, wherein the N input data can include input data of different categories. For example, input data A and data input B represent different categories, and N is equal to 2. The first feature matrix output by the N input data includes N*M feature values in total, wherein one input data corresponds to M channel feature values. In the first convolution process, the feature values can be understood as the original image data of the input, and in the subsequent convolution process, the feature values can also be intermediate data in the neural network operation process. In the convolution operation of the neural network, the feature values are also called feature maps, which carry feature data.

[0071] For example, taking the color image data belonging to an apple as an example, the apple is an input data, and the image data corresponding to the R channel, the G channel and the B channel in the color image are different feature values, and the number of feature values is 3.

[0072] In the first feature matrix, the N*M feature values are stored in the form of 1 row and N*M columns, and after conversion, the N*M feature values are stored in the form of N rows and M columns. One feature value can be understood as an element in the first feature matrix or the second feature matrix. Therefore, the first feature matrix can be understood as a one-dimensional storage format, that is, a format of 1 row and multiple columns, and the second feature matrix can be understood as a two-dimensional storage format, that is, a format of multiple rows and multiple columns.

[0073] In step S12 of the present disclosure, the storage format of the convolution matrix of O convolution kernels of O convolution layers is also converted before the convolution operation of the neural network, wherein one convolution layer includes M convolution kernels. Specifically, the first convolution matrix stored in the form of 1 row and O*M columns is converted into the second convolution matrix stored in the form of O rows and M columns, wherein one convolution kernel is an element in the first convolution matrix or the second convolution matrix.

[0074] It should be noted that in the neural network, one convolution layer can include M convolution kernels, which are one-to-one corresponding to the M channel feature values of one input data to perform convolution operation.

[0075] In step S13 of the present disclosure, the second feature matrix and the second convolution matrix after conversion are subjected to convolution operation, and a third feature matrix is obtained.

[0076] In an embodiment, the third feature matrix is also in a two-dimensional format. In order to facilitate the calculation after the convolution operation, the third feature matrix is restored to a one-dimensional format in the neural network. Therefore, in an embodiment, the data processing method further comprises:

[0077] converting the third feature matrix into a fourth feature matrix; wherein the third feature matrix comprises N rows and Q columns, and the Q feature values of the nth input data after convolution are located in the (n-1)*N+1 row to the nth*N row and the 1st to the Qth column, and Q is a positive integer; the fourth feature matrix comprises 1 row and N*Q columns, and the Q feature values of the nth input data after convolution are located in the (n-1)*Q+1 column to the nth*Q column of the fourth feature matrix.

[0078] It can be understood that the way of converting the third feature matrix into the fourth feature matrix is opposite to the way of converting the first feature matrix into the second feature matrix.

[0079] Figure 2 is a schematic diagram of the convolution process of the input data and the parameter data in the embodiment of the present disclosure, Figure 2 As shown, the original cache format is first converted into a two-dimensional image format, which specifically comprises: converting the cache format of the input data, i.e., the first feature matrix, to obtain the two-dimensional image format of the input data, i.e., the second feature matrix. The cache format of the parameter data, i.e., the first convolution matrix, is converted to obtain the two-dimensional image format of the parameter data, i.e., the second convolution matrix. The second feature matrix and the second convolution matrix after the format conversion are subjected to convolution operation to obtain a two-dimensional image format of the convolution result, which is the third feature matrix; and the two-dimensional image format of the convolution result is converted into the original cache format to obtain the convolution result in the original cache format, wherein the convolution result in the converted original cache format is the fourth feature matrix.

[0080] It should be noted that in the embodiment of the present disclosure, the convolution operation process of the converted second feature matrix and the second convolution matrix is still a process of dot product of the local region of the convolution kernel and the feature value. In the embodiment of the present disclosure, the convolution kernel is regarded as a whole, which is understood as an element in the second convolution matrix, and the feature value is regarded as a whole, which is understood as an element in the second feature matrix. The converted second convolution matrix and the second feature matrix can be understood as a two-dimensional matrix format, so that parallel computing can be used in the convolution operation process. For example, when the terminal supports GPU parallelism, the second feature matrix and the second convolution matrix are subjected to convolution operation by using the characteristics of GPU parallel computing.

[0081] Figure 3 is a schematic diagram of a convolution operation method, asFigure 3 As shown, convolution computation refers to performing a dot product operation on the input data and parameter data to obtain the output result. For example... Figure 3 As shown, the input data format is: height * width * number of channels, where the number of channels refers to the number of feature values ​​included in a single input data set, and height and width refer to the number of rows and columns of a single feature value, i.e., the number of rows and columns in the feature map. The feature values ​​carry the feature data. The parameter data format is: number of output channels * number of input channels * height * width. Here, the number of output channels refers to the number of convolutional layers, and the number of input channels refers to the number of convolutional kernels in a single convolutional layer, the same as the number of channels in the input data. The number of rows and columns of a convolutional kernel corresponds to the height and width in the parameter data, and the convolutional kernel carries the weight data.

[0082] like Figure 3 As shown, the input data consists of one data point, containing feature values ​​for three channels: in[:,:,0], in[:,:,1], and in[:,:,2]. The parameter data contains one convolutional layer, with three kernels: w[:,:,0], [:,:,1], and w[:,:,2]. The height and width of the feature values ​​are both 3, as are the height and width of the kernels. Based on the principle of convolution, the dot products of in[:,:,0] and w[:,:,0], in[:,:,1] and w[:,:,1], and in[:,:,2] and w[:,:,2] are taken, and the sum of these dot products yields the output matrix. The number of channels in the output matrix is ​​determined by the number of output channels in the parameter data, i.e., the number of convolutional layers. Therefore… Figure 2 The output matrix shown has a size of 1*1 and a channel count of 1.

[0083] In embodiments of this disclosure, a feature value is an element of a first feature matrix or a second feature matrix, and the feature value carries feature data. For example, if the feature value of one channel in the first feature matrix carries H1*W1 feature data, and the feature value of one channel in the second feature matrix carries H1*W1 feature data, and the second feature matrix includes (H1*N)*(W1*M) feature data; in one embodiment, the data processing method of this disclosure further includes:

[0084] When the product of W1*M is not a multiple of P, zeros are inserted at the end of the W1*M column of the second characteristic matrix to make the value of W1*M in the second characteristic matrix a multiple of P, where P is the number of bytes involved in one operation.

[0085] In the embodiment of the present disclosure, taking a 32-bit processor as an example, the processor reads and processes data blocks in units of 4 bytes, and therefore, to ensure the correctness of reading of feature data in the second feature matrix in the subsequent convolution operation, it is necessary to ensure that the value of W1*M is a multiple of 4.

[0086] In the embodiment of the present disclosure, one convolution kernel is one element in the first convolution matrix or the second convolution matrix, and the convolution kernel carries weight data. If one convolution kernel in the first convolution matrix carries H2*W2 weight data, step S12 includes:

[0087] When W2 is not equal to 1, the convolution kernel in the first convolution matrix is converted into a (H2*W2)*1 convolution kernel, and the second convolution matrix carrying (H2*W2*O)*M weight data is obtained.

[0088] In this embodiment, in the process of converting the first convolution matrix, each convolution kernel is also subjected to matrix transformation, and one convolution kernel is changed into a matrix of multiple rows and one column, and then mapped into the second convolution matrix, so as to adapt to the storage mode of the feature data in the second feature matrix, and ensure the correctness of the convolution operation.

[0089] According to the same principle of performing the data padding operation on the W1*M column feature data in the second feature matrix, in the embodiment of the present disclosure, when the product of H2*W2*O is not a multiple of P, 0 is inserted at the end of the H2*W2*O row of the second convolution matrix to pad, so that the value of (H2*W2*O) in the second convolution matrix is a multiple of P, wherein P is the number of bytes involved in one operation.

[0090] In an embodiment of the present disclosure, to improve the efficiency of data reading, the feature values in the second feature matrix are stored in groups, and the feature value group storage includes:

[0091] The feature values of the M channels are equally divided into r equal parts to form M / r feature value combinations;

[0092] One feature value in the second feature matrix after group storage carries H1*(W1*(M / r)) feature data, and the second feature matrix includes (H1*N)*(W1*(M / r)*r) feature data in total; wherein the channels of the feature values in one feature value combination are adjacent, and r is greater than 0; before the convolution operation, the stored second feature matrix is read in units of feature value combinations.

[0093] In this embodiment, M / r feature values ​​are stored as a combination. Therefore, the feature data corresponding to the coordinates of different combined feature values ​​is also a combination of M / r feature data. Thus, before performing convolution operations, the feature value combination is used as the reading unit. When obtaining feature data at a certain coordinate, M / r feature data can be read at once. It is understandable that this grouped storage method reduces the number of traversals and achieves fast access.

[0094] In one embodiment, in order to make full use of the characteristics of the processor reading data length and further improve the efficiency of data processing, the combination of feature values ​​can be set according to the number of bytes P involved in one operation of the processor, that is, the combination of P feature values, i.e., the value of M / r is equal to P.

[0095] For example, still taking a 32-bit processor as an example, the processor reads and processes data blocks in units of 4 bytes, so in the second feature matrix, 4 feature values ​​are stored as a group.

[0096] Figure 4 This is an example diagram illustrating the format conversion of input data to be convolved in an embodiment of this disclosure, such as... Figure 4 As shown, part A1 is the raw buffer format of the input data, such as... Figure 4 As shown, different feature values ​​of different input data are arranged sequentially. The original buffer format of the input data is NH1W1M, and the data volume carried by the buffer format is N1*H1*W1*M, where N is the batch size, representing the number of input data, H1 is the input height of the feature value, W1 is the input width of the feature value, and M is the number of channels, representing the number of feature values ​​corresponding to one input sample. It should be noted that in the embodiments of this disclosure, the feature value is treated as a whole. Figure 4 The original buffer in the data includes 1 row and N*M columns, corresponding to the first feature matrix of this embodiment.

[0097] like Figure 4 As shown, part B1 is the two-dimensional image (Image2D) format after the input data is converted. The Image2D format of the input data corresponds to the second feature matrix of this embodiment, as shown below. Figure 4 As shown, four adjacent channels are grouped together. Part B12 in the diagram represents a feature value combination with a height of H1 and a width of W1, carrying a total of H1*(W1*4) feature data. The element shown in B13 represents the four feature data combinations at corresponding coordinates for different feature values, originating from four different channels. Figure 3As shown, the height of the converted Image2D format is N*H1 and the width is W1*[(M+3) / 4], where W1*[(M+3) / 4] is used for 4-byte alignment.

[0098] In one embodiment of this disclosure, the convolution kernels in the second convolution matrix are stored in groups, using the same storage principle as the second feature matrix; wherein, the group storage of the convolution kernels includes:

[0099] Divide the O convolutional layers into s equal parts to form a combination of convolutional kernels for O / s convolutional layers;

[0100] The second convolutional matrix after group storage carries ((O / s)*H2*W2)*1 weight data in one convolutional kernel, and the second convolutional matrix includes a total of ((O / s)*H2*W2*s)*M ​​weight data; wherein, the convolutional layers to which the convolutional kernels belong within a convolutional kernel combination are adjacent, and s is greater than 0; before performing convolution operation, the stored second convolutional matrix is ​​read in units of convolutional kernel combinations.

[0101] In a preferred embodiment, to fully utilize the characteristics of the processor's data reading length and further improve the efficiency of data processing, the combination of convolution kernels can be set according to the number of bytes P involved in one operation of the processor, that is, P convolution kernels are combined, i.e., the value of O / s is equal to P.

[0102] Figure 5 This is an example diagram illustrating an exemplary format conversion of parameter data in an embodiment of this disclosure, such as... Figure 5 As shown, part A2 is the raw buffer format of the parameter data. The parameter data refers to the data of multiple convolutional kernels from multiple convolutional layers, such as... Figure 4 As shown, different convolutional kernels in different convolutional layers are arranged sequentially. The original buffer format of the convolutional kernels in different convolutional layers is: OMH2W2. The amount of data carried by the buffer format is: O*H2*W2*M, where O is the number of output channels, representing the number of convolutional layers, H2 is the height of a convolutional kernel, W2 is the width of a convolutional kernel, and M is the number of input channels, representing the number of convolutional kernels corresponding to one convolutional layer. It should be noted that in the embodiments of this disclosure, the convolutional kernel is treated as a whole. Figure 5 The original buffer in the data includes 1 row and 0*M columns, corresponding to the first convolution matrix in this embodiment of the present disclosure.

[0103] like Figure 5 As shown, part B2 is the Image2D format of the parameter data after conversion. The Image2D format of the parameter data corresponds to the second convolution matrix in this embodiment of the present disclosure, as shown below. Figure 5As shown, four adjacent convolutional layers are taken as a group, and the B22 part in the figure is a convolution kernel combination, the height of the convolution kernel combination is H2*W2, and the width is 1. The convolution kernel combination is formed by performing row and column conversion on the convolution kernel and then combining the converted convolution kernels from the four convolutional layers. One convolution kernel combination carries (4*H2*W2)*1 weight data. The element pointed to by B23 is a combination of four weight data in the convolution kernel combination at the corresponding coordinates of different convolution kernels, which come from different four convolutional layers. Figure 5 As shown, the height of the converted Image2D format is H2*W2*[(O+3) / 4], and the width is M, where H2*W2*[(O+3) / 4] is used for 4-byte alignment.

[0104] It can be understood that, in the embodiments of the present disclosure, the way of converting the first feature matrix into the second feature matrix and converting the first convolution matrix into the second convolution matrix will generate additional intermediate results or temporary auxiliary variables in memory storage when the template is slid by column and the data in the window is converted by row in the im2col mode. The data conversion mode of the embodiments of the present disclosure does not increase the additional storage space because it only involves matrix reorganization. The converted second feature matrix and second convolution matrix can facilitate parallel operation, and therefore the efficiency of convolution operation can be improved.

[0105] The following takes the neural network convolution operation applied in a mobile phone as an example, Figure 6 is an example diagram of a data processing method according to an embodiment of the present disclosure, as shown in Figure 6 The data processing method applied in the mobile phone includes the following steps:

[0106] S21, the mobile phone converts input data in a first format into a second format, where the first format is a one-dimensional storage format, and the second format is a two-dimensional storage format.

[0107] In this embodiment, the first format is Buffer format, and the second format is Image2D format. The input data in the first format is the first feature matrix in the present disclosure, and the input data in the second format is the second feature matrix.

[0108] S22, the mobile phone converts parameter data in the first format into the second format.

[0109] In this embodiment, the parameter data in the first format is the first convolution matrix in the present disclosure, and the parameter data in the second format is the second convolution matrix.

[0110] S23, the mobile phone performs convolution operation on the converted input data and parameter data in the second format by using a GPU to obtain a convolution result in the second format.

[0111] In this embodiment, the convolution result in the second format is the third feature matrix in the present disclosure.

[0112] For example, in the convolution operation process, a thread bundle is set to read feature data in the second feature matrix and weight data in the second convolution matrix in parallel by four threads, and four convolution results are output.

[0113] S24, the mobile phone converts the convolution result in the second format into the first format and outputs to the CPU.

[0114] In this embodiment, the converted convolution result in the first format is the fourth feature matrix in the embodiment of the present disclosure.

[0115] It can be understood that in the embodiment of the present disclosure, by taking advantage of the feature of GPU parallel computing supported by the mobile phone, when performing convolution operation of the neural network in the mobile phone, the input data and parameter data in the original CPU are converted into matrices in the same two-dimensional image format, and then the convolution operation is performed in parallel by using the GPU. On the one hand, no additional storage space is added, and on the other hand, the efficiency of the convolution operation is improved.

[0116] Figure 7 Fig. 1 is a diagram of a data processing apparatus according to an example embodiment. Referring to Fig. 1, Figure 7 The data processing apparatus comprises:

[0117] The first conversion module 101 is configured to convert a first feature matrix output by a neural network based on N input data into a second feature matrix; wherein the first feature matrix comprises 1 row and N*M columns; feature values of M channels of the nth input data are located in the (n-1)*M+1th column to the n*Mth column of the first feature matrix; n is a positive integer less than or equal to N, M is a positive integer, and the feature values of different channels refer to feature values obtained based on different feature extraction methods; the second feature matrix comprises N rows and M columns, wherein the M feature values of the nth input data are located in the (n-1)*N+1th row to the n*Nth row and the 1st to the Mth column.

[0118] The second conversion module 102 is configured to convert a first convolution matrix of the neural network into a second convolution matrix; wherein the first convolution matrix comprises 1 row and O*M columns; M convolution kernels of the oth convolution layer of the neural network are located in the (o-1)*M+1th column to the o*Mth column of the first convolution matrix; o is a positive integer less than or equal to O; the second convolution matrix comprises O rows and M columns, wherein the M convolution kernels of the oth convolution layer are located in the (o-1)*O+1th row to the o*Oth row and the 1st to the Mth column.

[0119] The convolution module 103 is configured to perform a convolution operation based on the second feature matrix and the second convolution matrix to obtain a third feature matrix.

[0120] In an embodiment, a feature value in one channel of the first feature matrix carries H1*W1 feature data; a feature value in one channel of the second feature matrix carries H1*W1 feature data, and the second feature matrix includes (H1*N)*(W1*M) feature data. The device further includes:

[0121] The first insertion module 104 is configured to pad 0 at the end of W1*M columns of the second feature matrix when the product of W1*M is not a multiple of P, so that the value of W1*M in the second feature matrix is a multiple of P, where P is the number of bytes involved in one operation.

[0122] In an embodiment, one convolution kernel in the first convolution matrix carries H2*W2 weight data, and the second conversion module 102 is specifically configured to convert the convolution kernel in the first convolution matrix into a (H2*W2)*1 convolution kernel when W2 is not equal to 1, to obtain the second convolution matrix carrying (H2*W2*O)*M weight data.

[0123] In an embodiment, the device further includes:

[0124] The second insertion module 105 is configured to pad 0 at the end of H2*W2*O rows of the second convolution matrix when the product of H2*W2*O is not a multiple of P, so that the value of (H2*W2*O) in the second convolution matrix is a multiple of P, where P is the number of bytes involved in one operation.

[0125] In an embodiment, the device further includes:

[0126] The first storage module 106 is configured to group the feature values in the second feature matrix, where the feature value grouping storage includes: equally dividing the feature values of the M channels into r equal parts to form M / r feature value combinations; one feature value in the second feature matrix after grouping storage carries H1*(W1*(M / r)) feature data, and the second feature matrix includes (H1*N)*(W1*(M / r)*r) feature data in total; where the channels of the feature values in one feature value combination are adjacent, and r is greater than 0.

[0127] The first reading module 107 is configured to read the stored second feature matrix in the feature value combination as a reading unit before performing the convolution operation.

[0128] In an embodiment, the apparatus further includes:

[0129] The second storage module 108 is configured to store the convolution kernel groups in the second convolution matrix; wherein the convolution kernel group storage includes: dividing O convolution layers into s equal parts to form O / s convolution kernel groups; one convolution kernel in the second convolution matrix after group storage carries ((O / s)*H2*W2)*1 weight data, and the second convolution matrix includes ((O / s)*H2*W2*s)*M weight data in total; wherein the convolution layers to which the convolution kernels in one convolution kernel group belong are adjacent in hierarchy, and s is greater than 0.

[0130] The second reading module 109 is configured to read the stored second convolution matrix in the convolution kernel group as a reading unit before performing the convolution operation.

[0131] In an embodiment, the apparatus further includes:

[0132] The third conversion module 110 is configured to convert the third feature matrix after convolution into a fourth feature matrix; wherein the third feature matrix includes: N rows and Q columns, Q feature values after convolution of the nth input data, located in the (n-1)*N+1 row to the nth*N row, and the 1st to the Qth column, Q being a positive integer; and the fourth feature matrix includes: 1 row and N*Q columns, Q feature values after convolution of the nth input data, located in the (n-1)*Q+1 column to the n*Q column of the fourth feature matrix.

[0133] As to the apparatus in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and will not be described in detail here.

[0134] Figure 8 is a block diagram of a terminal apparatus according to an exemplary embodiment. For example, the apparatus 800 can be a mobile phone, a mobile computer, or the like.

[0135] Referring to Figure 8 , the apparatus 800 can include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0136] The processing component 802 generally controls the overall operations of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions delivered from the memory 804 to complete all or part of the steps of the methods described above. In addition, the processing component 802 can include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0137] The memory 804 is configured to store various types of data to support the operations of the device 800. Examples of these data include instructions for any application or method operating on the device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0138] The power component 806 provides power to the various components of the device 800. The power component 806 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the device 800.

[0139] The multimedia component 808 includes a screen providing an output interface between the device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes the touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensors can not only sense a boundary of a touching or sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 808 includes a front camera and / or a back camera. The front camera and / or the back camera can receive external multimedia data when the device 800 is in an operating mode, such as a shooting mode or a video mode. Each of the front and back cameras can be a fixed optical lens system or have a focal length and optical zoom capability.

[0140] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive an external audio signal when the device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0141] The I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which can include a keypad, a click wheel, buttons, and so on. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0142] The sensor component 814 includes one or more sensors for providing status assessments of various aspects of the device 800. For example, the sensor component 814 can detect an open / closed position of the device 800, relative positioning of components, such as a display and a keypad of the device 800, a change of position of the device 800 or a component of the device 800, presence or absence of user contact with the device 800, changes in orientation or acceleration / deceleration

[0143] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a corresponding communication standard, such as Wi-Fi, 2G, or 3G, or a combination thereof. In an example embodiment, the communication component 816 receives broadcast signals or broadcast-related information from external broadcast management systems via a broadcast channel. In an example embodiment, the communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0144] In exemplary embodiments, the apparatus 800 can be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements for performing the above methods.

[0145] In exemplary embodiments, a non-transitory computer-readable storage medium including instructions, such as the memory 804 including instructions, is also provided, which can be executed by the processor 820 of the apparatus 800 to complete the above methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0146] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a terminal, enables the terminal to perform a control method, the method comprising:

[0147] convert a first feature matrix output by a neural network based on N input data into a second feature matrix; wherein the first feature matrix comprises: 1 row and N*M columns; feature values of M channels of the nth input data are located in the (n-1)*M+1th column to the n*Mth column of the first feature matrix; n is a positive integer less than or equal to N, M is a positive integer, and the feature values of different channels refer to feature values obtained based on different feature extraction methods; the second feature matrix comprises: N rows and M columns, wherein the M feature values of the nth input data are located in the (n-1)*N+1th row to the n*Nth row and the 1st to the Mth column;

[0148] convert a first convolution matrix of the neural network into a second convolution matrix; wherein the first convolution matrix comprises: 1 row and O*M columns; M convolution kernels of the oth convolution layer of the neural network are located in the (o-1)*M+1th column to the o*Mth column of the first convolution matrix; o is a positive integer less than or equal to O; the second convolution matrix comprises: O rows and M columns, wherein the M convolution kernels of the oth convolution layer are located in the (o-1)*O+1th row to the o*Oth row and the 1st to the Mth column;

[0149] perform a convolution operation based on the second feature matrix and the second convolution matrix to obtain a third feature matrix.

[0150] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the disclosure be construed as including any paterns of this disclosure that can be derived from the description and illustrations presented herein without departing from the scope and spirit of the disclosure. The specification and examples are exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0151] It is to be understood that the disclosure is not limited to the precise construction here described and illustrated and that various modifications and changes can be made without departing from the scope thereof. The only scope of the disclosure is to be determined by the appended claims.

Claims

1. A data processing method, characterized by, The method is applied to an electronic device, and the method comprises: Converting a first feature matrix output by a neural network based on N input data into a second feature matrix; wherein the first feature matrix comprises: 1 row and N*M columns; feature values of M channels of the nth input data are located in the (n-1)*M+1th column to the n*Mth column of the first feature matrix, and one channel of feature values in the first feature matrix carries H1*W1 feature data; n is a positive integer less than or equal to N, M is a positive integer, and feature values of different channels refer to feature values obtained based on different feature extraction methods; the input data comprises image data; the second feature matrix comprises: N rows and M columns, wherein M feature values of the nth input data are located in the (n-1)*N+1th row to the n*Nth row and the 1st to the Mth column, one channel of feature values in the second feature matrix carries H1*W1 feature data, and the second feature matrix comprises (H1*N)*(W1*M) feature data; Converting a first convolution matrix of the neural network into a second convolution matrix; wherein the first convolution matrix comprises: 1 row and O*M columns; M convolution kernels of the oth convolution layer of the neural network are located in the (o-1)*M+1th column to the o*Mth column of the first convolution matrix; o is a positive integer less than or equal to O; the second convolution matrix comprises: O rows and M columns, wherein M convolution kernels of the oth convolution layer are located in the (o-1)*O+1th row to the o*Oth row and the 1st to the Mth column; When the product of W1*M is not a multiple of P, 0 is inserted at the end of the W1*M columns of the second feature matrix to pad, so that the value of W1*M in the second feature matrix is a multiple of P, wherein P is the number of bytes involved in one operation of a GPU; Performing parallel convolution operation based on the second feature matrix and the second convolution matrix by using the GPU to obtain a third feature matrix.

2. The method of claim 1, wherein, One convolution kernel in the first convolution matrix carries H2*W2 weight data, and the conversion of the first convolution matrix of the neural network into the second convolution matrix comprises: When W2 is not equal to 1, the convolution kernel in the first convolution matrix is converted into a (H2*W2)*1 convolution kernel to obtain the second convolution matrix carrying (H2*W2*O)*M weight data.

3. The method of claim 2, wherein, The method further comprises: When the product of H2*W2*O is not a multiple of P, 0 is inserted at the end of the H2*W2*O rows of the second convolution matrix to pad, so that the value of (H2*W2*O) in the second convolution matrix is a multiple of P.

4. The method of claim 1, wherein, The method further comprises: The feature value grouping storage in the second feature matrix comprises: The M channel feature values are equally divided into r equal parts to form M / r feature value combinations; One feature value in the second feature matrix after grouping storage carries H1*(W1*(M / r)) feature data, and the second feature matrix includes (H1*N)*(W1*(M / r)*r) feature data in total; wherein the feature values in one feature value combination are adjacent in channel, and r is greater than 0; Before the convolution operation is performed, the second feature matrix after grouping storage is read as a reading unit.

5. The method of claim 3, wherein, The method further includes: The convolution kernels in the second convolution matrix are grouped and stored; wherein the convolution kernel grouping and storage includes: O convolution layers are equally divided into s equal parts to form O / s convolution kernel combinations; One convolution kernel in the second convolution matrix after grouping storage carries ((O / s)*H2*W2)*1 weight data, and the second convolution matrix includes ((O / s)*H2*W2*s)*M weight data in total; wherein the convolution layers to which the convolution kernels in one convolution kernel combination belong are adjacent in level, and s is greater than 0; Before the convolution operation is performed, the second convolution matrix after grouping storage is read as a reading unit.

6. The method of claim 1, wherein, The method further includes: The third feature matrix after convolution is converted into a fourth feature matrix; wherein the third feature matrix includes N rows and Q columns, Q feature values of the n input data after convolution, located in the (n-1)*N+1 row to the n*N row, and the 1st to the Qth column, and Q is a positive integer; and the fourth feature matrix includes 1 row and N*Q columns, Q feature values of the n input data after convolution, located in the (n-1)*Q+1 column to the n*Q column of the fourth feature matrix.

7. A data processing apparatus, characterized by, The device includes: A first conversion module configured to convert a first feature matrix output by a neural network based on N input data into a second feature matrix; wherein the first feature matrix includes 1 row and N*M columns; M channel feature values of the n input data, located in the (n-1)*M+1 column to the n*M column of the first feature matrix, and one channel feature value in the first feature matrix carries H1*W1 feature data; n is a positive integer less than or equal to N, M is a positive integer, and different channel feature values refer to feature values obtained based on different feature extraction methods; the input data includes image data; and the second feature matrix includes N rows and M columns, wherein M feature values of the n input data are located in the (n-1)*N+1 row to the n*N row, and the 1st to the Mth column, one channel feature value in the second feature matrix carries H1*W1 feature data, and the second feature matrix includes (H1*N)*(W1*M) feature data in total; The second conversion module is configured to convert the first convolution matrix of the neural network into a second convolution matrix; wherein the first convolution matrix comprises 1 row and O*M columns; M convolution kernels of an oth convolution layer of the neural network are located in (o-1)*M+1th column to o*Mth column of the first convolution matrix; o is a positive integer less than or equal to O; the second convolution matrix comprises O rows and M columns, wherein M convolution kernels of the oth convolution layer are located in (o-1)*O+1th row to o*Oth row and 1st column to Mth column. The first insertion module is configured to insert 0 at the end of W1*M columns of the second feature matrix to pad up to make the value of W1*M in the second feature matrix a multiple of P when the product of W1*M is not a multiple of P, wherein P is the number of bytes involved in one operation of the GPU; The convolution module is configured to perform parallel convolution operation by using the GPU based on the second feature matrix and the second convolution matrix to obtain a third feature matrix.

8. The apparatus of claim 7, wherein, One of the convolution kernels in the first convolution matrix carries H2*W2 weight data, The second conversion module is specifically configured to convert the convolution kernel in the first convolution matrix into a (H2*W2)*1 convolution kernel to obtain the second convolution matrix carrying (H2*W2*O)*M weight data when W2 is not equal to 1.

9. The apparatus of claim 8, wherein, The device further comprises: The second insertion module is configured to insert 0 at the end of H2*W2*O rows of the second convolution matrix to pad up to make the value of H2*W2*O in the second convolution matrix a multiple of P when the product of H2*W2*O is not a multiple of P.

10. The apparatus of claim 7, wherein, The device further comprises: The first storage module is configured to group store feature values in the second feature matrix, wherein the group store of the feature values comprises: equally dividing the feature values of the M channels into r equal parts to form M / r feature value combinations; one feature value in the group stored second feature matrix carries H1*(W1*(M / r)) feature data, and the second feature matrix comprises (H1*N)*(W1*(M / r)*r) feature data in total; wherein the channels of the feature values in one of the feature value combinations are adjacent, and r is greater than 0; The first reading module is configured to read the stored second feature matrix in the feature value combination as a reading unit before the convolution operation is performed.

11. The apparatus of claim 9, wherein, The device further comprises: The second storage module is configured to group store convolution kernels in the second convolution matrix; wherein the group store of the convolution kernels comprises: equally dividing O convolution layers into s equal parts to form O / s convolution kernel combinations of the convolution layers; one convolution kernel in the group stored second convolution matrix carries ((O / s)*H2*W2)*1 weight data, and the second convolution matrix comprises ((O / s)*H2*W2*s)*M weight data in total; wherein the levels of the convolution layers to which the convolution kernels in one of the convolution kernel combinations belong are adjacent, and s is greater than 0; The second reading module is configured to read the stored second convolution matrix as a reading unit in combination with the convolution kernel before the convolution operation is performed.

12. The apparatus of claim 7, wherein, The device further comprises: The third conversion module is configured to convert the third feature matrix after the convolution into a fourth feature matrix; wherein the third feature matrix comprises N rows and Q columns, Q feature values of the input data after the convolution, and the Q feature values are located in the (n-1)*N+1th row to the n*Nth row and the 1st to the Qth column, Q is a positive integer; the fourth feature matrix comprises 1 row and N*Q columns, Q feature values of the input data after the convolution, and the Q feature values are located in the (n-1)*Q+1th column to the n*Qth column of the fourth feature matrix.

13. A terminal, characterized by comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the data processing method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium, comprising: When the instructions in the storage medium are executed by the processor of the terminal, the terminal can perform the data processing method according to any one of claims 1 to 6.

Images