A convolutional neural network computing structure and application method

Abstract

The application provides a convolutional neural network structure and an application method. The calculation structure comprises at least N convolutional layers, and the N convolutional layers are sequentially connected. The N convolutional layers comprise a first convolutional layer and a calculation convolutional layer. The first convolutional layer is used for convolutional calculation on an input data set of the input layer. The calculation convolutional layer comprises N-1 convolutional layers. The calculation convolutional layer is used for convolutional calculation on a calculation result of a previous convolutional layer connected with the current convolutional layer and historical data of a next layer. By adjusting the positions of the convolutional and pooling layers, a plurality of learnable data parameters are added at the end of the initial input data, so that the dimension of the final data output meets the expectation. In this way, by adjusting the calculation structure of a general convolutional neural network model, the calculation graph logic of each calculation stage in the streaming inference process is ensured to be consistent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural networks, in particular to a convolutional neural network computing structure and an application method. BACKGROUND

[0002] The initial main use scenario of the convolutional neural network is a task in the field of graphics and images, and the main use requirement in the application scenario is offline input and calculation, that is, the task to be calculated is input into the model once, and the output result is waited.

[0003] However, as the structure of the convolutional neural network becomes more and more complex, the expression ability of the model is enhanced, and it is gradually used in real-time computing scenarios such as speech recognition. The training process of this type of task does not change much compared to the traditional training scheme, and the training process of inputting the whole sentence offline into the model can still be used. This training method has a positive effect on accelerating the training process and better utilizing the computing power of hardware.

[0004] However, this method introduces deployment problems at the inference end. In actual inference tasks, there is a certain requirement for the response time of task calculation, which can cause a lot of hardware delay. Secondly, offline input requires high computing burst capability of the computing device. After the whole sentence input is completed, the computing result needs to be given as soon as possible, and the computing device is relatively idle during other time periods, which can cause waste of computing resources. Therefore, the currently commonly used method is to input data in a streaming manner for calculation, and the data input before the input is completed is input into the computing device for calculation. Taking speech as an example, 10ms is generally taken as a basic unit, and the specific input length is mainly based on the actual model structure. In the process of streaming input, in order to avoid repeated calculation, the history data copying method is usually used for data reuse, but it can cause inconsistency of copied history and future data in different calculation stages (input start, middle input, and input end). Under the current static graph inference engine, three inference graphs are needed to realize the whole streaming process, which can cause great scheduling overhead and waste of calculation. SUMMARY

[0005] Therefore, the present application provides a convolutional neural network computing structure and an application method, which aims to avoid the problem of inconsistency of the inference network caused by history copying and right data pad in the original convolution and deconvolution calculation process, and greatly simplifies the inference implementation of the streaming network.

[0006] In a first aspect, the present application provides a convolutional neural network computing structure, characterized in that the computing structure comprises at least N convolutional layers, and the N convolutional layers are connected in sequence.

[0007] The N convolutional layers include a first convolutional layer and N-1 convolutional layers, the first convolutional layer is configured to perform convolution calculation on the input data set of the input layer, and the N-1 convolutional layers are configured to perform convolution calculation on the calculation result of the previous convolutional layer connected with the current convolutional layer and the history data of the next layer.

[0008] Optionally, the computing structure further includes at least one pooling layer, the N is an even number not less than 2, and the input layer, the convolutional layers, the pooling layer and the output layer are sequentially connected, and the pooling layer is configured to perform pooling on the calculation result of the Nth convolutional layer to output the final result.

[0009] Optionally, the input data set includes original input data and input layer supplementary data, the input layer supplementary data includes first side supplementary data and second side supplementary data, the first side supplementary data is supplementary data corresponding to the first data of the original input data in the input data set, and the second side supplementary data is supplementary data corresponding to the last data of the original input data in the input data set.

[0010] Optionally, the number of the first side supplementary data is determined according to the dimension of the convolution kernel, and the number of the second side supplementary data is determined according to the number of the convolutional layers and is used for supplementing the calculation result.

[0011] Optionally, the first convolutional layer is configured to take each data in a first set as a first layer calculation reference, take x data before each first layer calculation reference as history data, perform convolution operation on each first layer calculation reference and the history data corresponding to the first layer calculation reference, and take the operation result as the first layer calculation result, the x is the number of the first side supplementary data, and the first set is a data set sequentially composed of the first data of the original input data to the last data of the input data set.

[0012] Optionally, the first convolutional layer to the Nth convolutional layer are sequentially connected.

[0013] The Nth convolutional layer is configured to take each data in an Nth set as a next layer calculation reference, take x data before each next layer calculation reference as history data, perform convolution operation on each next layer calculation reference and the history data corresponding to the next layer calculation reference, and take the operation result as the Nth layer calculation result, the Nth set is a data set sequentially composed of the first data of the N-1th layer calculation result to the last data of the input data set.

[0014] Optionally, the final result is obtained by the pooling layer merging a plurality of data in the Nth layer calculation result, the Nth layer calculation result includes dirty data and valid data, the dirty data is calculated by taking x supplementary data as historical data, and the number of dirty data is determined according to the number N of convolution layers.

[0015] In a second aspect, the present application provides a data processing method based on a convolutional neural network calculation structure, the method comprising:

[0016] performing convolution calculation on an input data set of an input layer by a first convolution layer of the neural network calculation structure;

[0017] performing convolution calculation on a calculation result of a previous convolution layer connected to the current convolution layer and historical supplementary data of a next layer by a calculation convolution layer.

[0018] Optionally, the input data set includes original input data and input layer supplementary data, the input layer supplementary data includes first side supplementary data and second side supplementary data, the first side supplementary data is supplementary data corresponding to the first data of the original input data in the input data set, and the second side supplementary data is supplementary data corresponding to the last data of the original input data in the input data set.

[0019] Optionally, the convolution calculation on the input data set of the input layer by the first convolution layer of the convolutional neural network structure comprises:

[0020] obtaining each data in the first set as a first layer calculation reference;

[0021] obtaining x data before each first layer calculation reference as historical data;

[0022] performing convolution operation on each first layer calculation reference and the historical data corresponding to the first layer calculation reference to determine an operation result as a first layer calculation result, x is the number of the first side supplementary data, and the first set is a data set composed of the first data of the original input data to the last data of the input data set in turn.

[0023] Optionally, the convolution calculation on the calculation result of the previous convolution layer connected to the current convolution layer by the calculation convolution layer comprises:

[0024] obtaining each data in the Nth set as a next layer calculation reference;

[0025] obtaining x data before each calculation reference as historical data;

[0026] The convolution operation is performed on the basis of each layer calculation reference and the historical data corresponding to the layer calculation reference, and the operation result is the Nth layer calculation result, and the Nth set is a data set composed of the first data of the N-1th layer calculation result to the last data of the input data set.

[0027] The application provides a convolutional neural network computing structure and an application method. The computing structure comprises at least N convolutional layers, and the N convolutional layers are sequentially connected; the N convolutional layers comprise a first convolutional layer and a calculation convolutional layer, the first convolutional layer is used for performing convolution calculation on an input data set of the input layer, and the calculation convolutional layer comprises N-1 convolutional layers, and the calculation convolutional layer is used for performing convolution calculation on a calculation result of a last convolutional layer connected to the current convolutional layer and layer historical data; by adjusting the positions of the convolutional and pooling layers, the pooling process is ensured, the data processed each time is dirty data or valid data, and the pooling process between the dirty data and the valid data does not occur, so that the pollution of the dirty data and the valid data can be avoided. In addition, a plurality of learnable data parameters are added at the end of the input data at the beginning, so that the dimension of the final data output meets the expectation, and thus, by adjusting the computing structure of a general convolutional neural network model, the logic consistency of the computing graph in each computing stage in the streaming inference process is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0028] To make the technical solutions in the embodiments or the prior art clearer, the accompanying drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the application, and other accompanying drawings can be obtained by those skilled in the art without any creative effort.

[0029] Figure 1 A streaming computing process schematic diagram of a convolution operator in the prior art is provided for the embodiments of the application.

[0030] Figure 2 A streaming computing schematic diagram of a deconvolution operator in the prior art is provided for the embodiments of the application.

[0031] Figure 3 A convolutional neural network computing structure is provided for the embodiments of the application.

[0032] Figure 4 A convolutional streaming reforming structure after parameter configuration is provided for the embodiments of the application.

[0033] Figure 5 A deconvolutional streaming reforming structure after parameter configuration is provided for the embodiments of the application. DETAILED DESCRIPTION

[0034] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the accompanying drawings in the embodiments of the present application, so that those skilled in the art can better understand the solutions in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0035] As described above, the main use scenario of the convolutional neural network is the task in the field of graphics and images, and the main use requirement is offline input and calculation, that is, the task to be calculated is input into the model once, and the output result can be obtained. However, the inventors have found that the method commonly used at present is to input data for calculation in a streaming manner, and the data input is not completed before the data input before is sent to the calculation device for calculation. For example, 10 ms is generally used as a basic unit for voice, and the specific input length is mainly based on the actual model structure. In the process of streaming input, the convolution and deconvolution operators usually use the historical data copy method for data reuse to avoid repeated calculation, but this will cause the copied historical and future data to be inconsistent in different calculation stages, which will cause three inference graphs to be required to realize the entire streaming process under the current static graph inference engine.

[0036] Figure 1 The existing technology provides a flow calculation process diagram of a convolution operator for the embodiments of the present application, that is, the flow calculation process diagram of the convolution operator in the prior art. The input is sent in three times, 16 data (virtual node-1 represents the pad data of the convolution operator) are sent at a time, the data 0 to 10 are input for the first time, the output data 0 to 3 are calculated, the input data 11 to 14 are input for the second time, the output data 4 and 5 are calculated, the input data 15 is input for the third time, and the output data 6 and 7 are obtained.

[0037] In the first stage, each convolution operator only needs to do normal left pad, i.e. supplementing the historical data dimension by 1; in the second calculation of inputs 11 to 14, in order to save the calculation amount, for conv0, the 9th and 10th input nodes are needed as historical data to cooperate with the 11th to 14th to complete the calculation of conv0, and for conv1, the 3rd and 4th nodes of the pool0 output are needed as historical nodes to cooperate with the 5th and 6th nodes of the new pool0 output to complete the calculation of conv1, i.e. the second stage supplements the historical data dimension by 2; in the third stage, in addition to the same as the second stage, the supplement of the historical data dimension is 2, the original convolution needs to be supplemented with right pad data, i.e. the rightmost -1 node. In the three different calculation stages of convolution, the dimension ratio relationship of the input and output data is inconsistent, and the supplement dimensions of the historical and future data are also inconsistent, so for the inference engine of a static graph, three different inference networks need to be used to realize it.

[0038] Figure 2 The prior art deconvolution operator flow calculation diagram provided for the embodiments of the present application is a flow calculation diagram of a deconvolution operator in the prior art. If the deconvolution operator parameters are kernel=4, stride=2, padding=1 and outpadding=0, the flow calculation process is as follows. In the first stage, input the 0th data, and output the 0th node after deconvolution calculation; in the second stage, input the 1st and 2nd data, and output the 1st to 4th nodes after deconvolution calculation; in the third stage, input the end node 3rd data, and output the 5th to 7th nodes after deconvolution calculation. In the first stage, one input node can be deconvolved to obtain one output node; in the second stage, the last node of the previous stage is needed as historical data, i.e. the 0th node, which is combined with the current inputs 1st and 2nd to be input into the calculation; in the third stage, one historical data node is supplemented as in the second stage, but due to the right pad, the data ratio of the output node is more than that in the second stage.

[0039] It should be noted that kernel represents the dimension of the convolution kernel, pad represents the boundary supplement of convolution, and stride represents the step length of convolution calculation. Here, the distance about the prior art is the most commonly used configuration scheme of convolution. These parameters can be modified to other parameters, which are not limited in the present application.

[0040] As can be seen from the above, in the three calculation stages of the flow deconvolution, the dimension relationship of the input and output data is different, and the historical data supplement dimensions of each stage are also different, which leads to the need for three different calculation graphs to realize the inference process, thereby increasing the complexity of the inference deployment process.

[0041] To solve the above problems, an embodiment of the present application provides a streaming training scheme for a neural network model involving convolution and deconvolution, and the inference process is realized by only one image through modification of the neural network model.

[0042] Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0043] Referring to Figure 3 , Figure 3 A convolutional neural network computing structure provided by an embodiment of the present application comprises:

[0044] At least N convolutional layers, which are sequentially connected.

[0045] The N convolutional layers in the structure comprise a first convolutional layer and a calculation convolutional layer.

[0046] The first convolutional layer is configured to perform convolutional calculation on an input data set of the input layer, and the calculation convolutional layer comprises N-1 convolutional layers, and the calculation convolutional layer is configured to perform convolutional calculation on a calculation result of a previous convolutional layer connected to the current convolutional layer.

[0047] In an actual application scenario, convolutional operators can be directly connected. For a network comprising a pooling layer, there is a certain limitation on the number of layers. The computing structure further comprises at least one pooling layer, in which case the N is limited to an even number not less than 2. The structure is sequentially connected in the order of the input layer, the convolutional layers, the pooling layer and the output layer. The pooling layer is configured to perform pooling on a calculation result of the Nth convolutional layer, so as to output a final result.

[0048] Figure 4 A parameter-configured convolutional streaming modification structure provided by an embodiment of the present application is described below with reference to the network structure of the prior art conv-pool-conv. As shown in FIG. 2, after streaming modification, the dimension ratio of input and output data can be kept as 2:1 in different calculation stages in the streaming calculation process. Figure 4

[0049] Figure 4 The leftmost -2 to the rightmost -2 part in the first row in FIG. 2 is an input data set of the input layer in the structure.

[0050] ​The input data set includes original input data and input layer supplementary data, the input layer supplementary data includes first side supplementary data and second side supplementary data, the first side supplementary data is supplementary data corresponding to the first data of the original input data in the input data set, and the second side supplementary data is supplementary data corresponding to the last data of the original input data in the input data set.

[0051] Figure 4 The original input data is 0 to 15, the first data of the original input data is 0, the last data of the original input data is 15, the first side supplementary data is -1 and -2 on the left, and the second side supplementary data is -1 and -2 on the right. It should be noted that the dirty data according to the first side supplementary data needs to be discarded in the final calculation, and in order to ensure that the total amount of data calculated finally is correct, the input needs to be supplemented on the right, that is, the second side supplementary data is supplemented on the right to fill the output.

[0052] The number of first side supplementary data is determined according to the dimension of the convolution kernel, and the number of second side supplementary data is determined according to the number of convolution layers and used for supplementing the calculation result. For example Figure 4 In the example, the first side supplementary data is 2, the number of first side supplementary data = 3 (the dimension of the convolution kernel) - 1, the number of second side supplementary data on the right is 2, and in the current example, there are two convolution layers, and the number of second side supplementary data = the number of convolution layers.

[0053] The convolution operation process of each convolution layer is described as follows:

[0054] The first convolution layer is used to calculate the first layer calculation reference with each data in the first set as the first layer calculation reference, use x data before each first layer calculation reference as historical data, and combine each first layer calculation reference and the historical data corresponding to the first layer calculation reference to perform convolution operation, and the operation result is the first layer calculation result, x is the number of first side supplementary data, and the first set is a data set composed of the first data of the original input data to the last data of the input data set in turn.

[0055] Figure 4 In the example, 0 to -2 is the first set, -1 and -2 on the left are historical data, that is, the first set includes the original input data and the second side supplementary data. The second side supplementary data is to add a number of learnable data parameters at the end of the initial input data, so that the output dimension of the last data meets the expectation, Figure 4The supplementary data in the example is 2, that is, the -1 and -2 nodes on the rightmost side. In an actual application scenario, 0 can use -1 and -2 as historical data for convolution calculation, 0 is a calculation reference, -1 and -1 data nodes are historical data, and a calculation result is obtained through convolution operation, that is, -1 in the next layer. The data in the first set in the first convolution layer is respectively subjected to convolution operation in this way, and a first layer calculation result is obtained, that is, Figure 4 -1 on the left side to -1 on the right side of the second layer.

[0056] Further, the structure includes N convolution layers, for example, when N = 4, an order connection of an input layer-first convolution layer-second convolution layer-third convolution layer-fourth convolution layer-pooling layer-output layer can be used.

[0057] The Nth convolution layer is used to take each data in the Nth set as a next layer calculation reference, take x data before each calculation reference as next layer historical data, and combine each next layer calculation reference and the corresponding historical data to perform convolution operation, and an operation result is an Nth layer calculation result. The Nth set is a data set composed of data from the first data of the N-1th layer calculation result to the last data of the input data set. That is, the number of historical supplementary data on the left side of each convolution layer is the same, and the supplementary historical data has consistent dimensions. In an actual application scenario, the next layer supplementary data and the first side supplementary data can be supplemented as 0, the second side supplementary data can be supplemented as a constant 0, a constant 1, or data of 15 data nodes is copied.

[0058] In Figure 4 In the example, N = 2, so the second convolution layer is used to calculate according to the calculation result of the first convolution layer and corresponding next layer historical data. The next layer historical data is x data (x value is 2 in the example) before the first data of the upper layer calculation result, which is next layer historical data. Here, x is the number of first side supplementary data. Figure 4 In the example, -1 of the second layer is calculated by the convolution of the first row, -2 and -3 have similar meanings to -1 and -2 of the first row, and are supplementary historical data, that is, the next layer historical data described in the claims. After the convolution of the first row is completed, the dirty data -1 of the second row is generated. After the convolution of the second row is completed, the dirty data -1 and -2 of the third row is generated.

[0059] Regarding the final result obtained by calculation in the calculation structure, the relevant content is as follows:

[0060] The final result is obtained by the pooling layer combining a plurality of data in the Nth layer calculation result. The Nth layer calculation result includes dirty data and valid data. The dirty data is calculated by taking x supplementary data as historical data. The Nth layer calculation result is a calculation result obtained by the Nth convolution layer performing convolution calculation according to the calculation result of the N-1th layer and the historical data of the N-1th layer. For example, in Figure 4 In the example, the second layer calculation result is obtained by performing convolution calculation on the first layer calculation result and the historical data of the second layer. The historical data of the second layer is -2 and -3 in the figure. After the convolution calculation of the first row is completed, the dirty data -1 of the second row is generated. After the convolution calculation of the second row is completed, the dirty data -1 and -2 of the third row are generated.

[0061] In an actual application scenario, the number of frames of dirty data that can be calculated is related to the number of convolution layers. Taking a convolution with a kernel of 3 as an example, one dirty data is additionally generated for each additional convolution layer. In this embodiment, the dirty data is all located on the left. The number of dirty data is determined according to the foregoing. The number of dirty data increases with the number of convolution layers. After the pool layer, two dirty data become one dirty data, that is, -1 of the fourth row.

[0062] As described above, increasing the number of convolution layers increases the number of dirty data on the left. The pool layer can combine a plurality of data into one data. To prevent interference of dirty data, the data calculated by the pool layer should be all clean data or all dirty data. Otherwise, the clean data and the dirty data may interfere with each other. If the step of the pool layer is 2, the convolution 1, the convolution 2, and the pool layer should be performed. Otherwise, the clean data and the dirty data may interfere with each other when the pool layer is performed.

[0063] In an actual application scenario, the number of pool layers is related to the number of dirty data. The pool layer is used to combine a plurality of data. In an application, the number of pool layers can be set according to actual needs. For example, when the number of convolution layers or the number of dirty data is large, a corresponding pool layer is arranged after a plurality of convolution layers to combine and process the data. The convolution layer can be connected and subsequent operations can be performed on the basis. The number of pool layers can be adjusted according to actual needs, which is not limited in this application.

[0064] The last output layer. The first plurality of outputs are dirty data, which can be discarded in the calculation process. In the example, there is one dirty data. This ensures that the dimension of the output data is consistent with the original, and avoids interference of the dirty data at the beginning on the calculation process. The historical data of the convolution is all 2, which avoids the problem in the original scheme, that is, the dimension of the historical data is 1 in the first stage, and the dimension of the historical data is 2 in the second and third stages.

[0065] Relatedly, Figure 5A parameter configured deconvolution stream reconstruction structure is provided for an embodiment of the present application. The deconvolution stream reconstruction is shown in Figure 5 The deconvolution process represents a process from less to more, and a supplementary data node is added to the left of the first data in the original input data to ensure that the field of view does not change during the calculation process.

[0066] Figure 5 In an example, the original deconvolution parameters can be set as kernel=4, stride=2, padding=1, outpadding=0, the stream reconstruction changes the padding to 2, and a virtual learnable parameter node -1 is added to the leftmost node of the original input data. After the reconstruction, in the first stage, the input of the 0th valid data node can obtain the output data 0 and 1; in the second stage, the input of the 1st and 2nd valid data nodes can obtain the output data 2 to 5; in the third stage, the input of the 7th valid data node can obtain the output data 6 and 7. The input-output ratio is 1:2, and the three stages remain consistent. In the inference process, in each stage, a historical node is added to the left, and the inference graph description is consistent.

[0067] After the above reconstruction of the convolution operator, the dimension ratio of the input and output data can be ensured to be consistent in the inference process, the dimension of the supplementary historical data is consistent, the right pad problem of the original third stage is solved, and the inference network is unique. For real-time stream computing tasks involving convolution and deconvolution operators, the problem of inconsistent inference network caused by historical copying and right data pad in the original convolution and deconvolution calculation process can be avoided, which greatly simplifies the inference implementation of the stream network, especially for cloud multi-path computing stream tasks. Differentiated processing of different computing stages will cause great scheduling overhead and waste of calculation. In actual application scenarios, the network structure may need to be adjusted, which will introduce changes in effect and efficiency, and the network needs to be trained again, which needs to be paid attention to in the network design stage.

[0068] According to an embodiment of the present application, a data processing method based on a convolution neural network computing structure is provided,

[0069] The input data set of the input layer is convolved by the first convolution layer of the neural network computing structure;

[0070] The calculation result of the previous convolution layer connected to the current convolution layer and the next layer historical supplementary data are convolved by the calculation convolution layer.

[0071] Optionally, the input data set comprises original input data and input layer supplementary data, the input layer supplementary data comprises first side supplementary data and second side supplementary data, the first side supplementary data is supplementary data corresponding to the first data of the original input data in the input data set, and the second side supplementary data is supplementary data corresponding to the last data of the original input data in the input data set.

[0072] Optionally, the convolution calculation of the input data set of the input layer by the first convolution layer of the convolution neural network structure comprises:

[0073] The first set of data is obtained as the first layer calculation reference.

[0074] The x data before each first layer calculation reference is obtained as historical data.

[0075] The convolution operation is performed in combination with each first layer calculation reference and the historical data corresponding to the first layer calculation reference, and the operation result is determined as the first layer calculation result, the x is the number of the first side supplementary data, and the first set is a data set composed of the first data of the original input data to the last data of the input data set in turn.

[0076] Optionally, the convolution calculation of the calculation result of the previous convolution layer connected to the current convolution layer by the calculation convolution layer comprises:

[0077] The Nth set of data is obtained as the second layer calculation reference.

[0078] The x data before each calculation reference is obtained as historical data.

[0079] The convolution operation is performed in combination with each second layer calculation reference and the second layer historical data corresponding to the second layer calculation reference, and the operation result is determined as the Nth layer calculation result, the Nth set is a data set composed of the first data of the (N-1)th layer calculation result to the last data of the input data set in turn, and the first convolution layer is sequentially connected to the Nth convolution layer.

[0080] Optionally, the pooling of the calculation result of the Nth layer convolution layer by the pooling layer and the output of the final result comprise:

[0081] The final result is obtained by the pooling layer from a plurality of data in the Nth layer calculation result, the Nth layer calculation result comprises dirty data and valid data, the dirty data is calculated by taking x supplementary data as historical data, the Nth layer calculation result is a calculation result obtained by the Nth convolution layer from the convolution calculation of the calculation result of the (N-1)th layer and the second layer supplementary data of the (N-1)th layer, and the number of dirty data is determined according to the number N of convolution layers.

[0082] Compared with the common convolution scheme existing in the prior art, if kernel = 3, when calculating the convolution, the current input needs to see one input on the left and one input on the right, and then the calculation is performed. We have modified this convolution operator to use two inputs on the left plus the current input to perform the calculation. Each step of calculation only uses two on the left plus the current one, unlike the previous one that uses one on the left, one on the right and one on the current. For real-time streaming computing tasks of neural networks involving convolution and deconvolution operators, the inconsistency problem of inference networks caused by historical copying and right data pad in the original convolution and deconvolution calculation process can be avoided, greatly simplifying the inference implementation of the streaming network, especially the streaming task of cloud multi-path calculation. Differentiated processing in different calculation stages will cause great scheduling overhead and waste of calculation.

[0083] The embodiment of the application also provides a data processing method based on a convolution neural network computing structure, which comprises the following steps: performing convolution calculation on an input data set of an input layer by a first convolution layer of a neural network computing structure; and performing convolution calculation on a calculation result of a previous convolution layer connected with the current convolution layer and next layer historical supplementary data by the calculation convolution layer.

[0084] Optionally, the input data set comprises original input data and input layer supplementary data, the input layer supplementary data comprises first side supplementary data and second side supplementary data, the first side supplementary data is supplementary data corresponding to first data of the original input data in the input data set, and the second side supplementary data is supplementary data corresponding to last data of the original input data in the input data set.

[0085] Optionally, the computing structure further comprises at least one pooling layer, N is an even number not less than 2, and the calculation result of the Nth convolution layer is pooled by the pooling layer to output a final result.

[0086] Optionally, the number of the first side supplementary data is determined according to the dimension of the convolution kernel, and the number of the second side supplementary data is determined according to the number of convolution layers and is used for supplementing the calculation result.

[0087] Optionally, the convolution calculation on the input data set of the input layer by the first convolution layer of the convolution neural network structure comprises the following steps:

[0088] Each data in the first set is obtained as a first layer calculation reference;

[0089] x data before each first layer calculation reference are obtained as historical data;

[0090] The convolution operation is performed on each first layer calculation reference and historical data corresponding to the first layer calculation reference, and a calculation result is determined as a first layer calculation result, x is the number of the first side supplementary data, and the first set is a data set formed by the first data of the original input data to the last data of the input data set in sequence.

[0091] Optionally, the convolution calculation performed by the calculation convolution layer on the calculation result of the previous convolution layer connected to the current convolution layer includes:

[0092] Each data in the Nth set is obtained as a second layer calculation reference.

[0093] x data before each calculation reference is obtained as historical data.

[0094] The convolution operation is performed on each second layer calculation reference and historical data corresponding to the second layer calculation reference, and a calculation result is determined as an Nth layer calculation result, the Nth set is a data set formed by the first data of the (N-1)th layer calculation result to the last data of the input data set in sequence, and the first convolution layer is sequentially connected to the Nth convolution layer.

[0095] Optionally, the method further includes:

[0096] A final result is obtained, the final result is obtained by the pooling layer by merging a plurality of data in the Nth layer calculation result, the Nth layer calculation result includes dirty data and valid data, the dirty data is calculated by taking x supplementary data as historical data, the Nth layer calculation result is a calculation result obtained by the Nth convolution layer by performing convolution calculation on the calculation result of the (N-1)th layer and the second layer supplementary data of the (N-1)th layer, and the number of the dirty data is determined according to the number N of convolution layers.

[0097] Embodiments of the present application also provide a corresponding device and a computer storage medium for implementing the scheme provided by the embodiments of the present application.

[0098] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes to enable the device to execute the data processing method based on the convolution neural network calculation structure according to any embodiment of the present application.

[0099] The computer storage medium stores codes, and when the codes are executed, a device executing the codes implements the data processing method based on the convolution neural network calculation structure according to any embodiment of the present application.

[0100] The "first" and "second" in the names mentioned in the embodiments of the present application are only used for name identification, and do not represent the first and second in order.

[0101] From the description of the above embodiments, those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solutions of the present application can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of the present application.

[0102] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, they are described more simply, and the relevant parts can be referred to the part of the description of the method embodiments. According to the actual needs, some or all of the modules can be selected to achieve the purpose of the embodiment. Those skilled in the art can understand and implement without creative labor.

[0103] The above only describes exemplary embodiments of the present application, and is not intended to limit the protection scope of the present application.

Claims

1. A convolutional neural network computational structure, applied to real-time streaming data processing scenarios, wherein the real-time streaming data processing scenario includes speech recognition, characterized in that, The computational structure includes: at least N convolutional layers, wherein the N convolutional layers are connected sequentially; The N convolutional layers include a first convolutional layer and a computational convolutional layer. The first convolutional layer is used to perform convolutional calculations on the input dataset of the input layer. The computational convolutional layer includes N-1 convolutional layers. The computational convolutional layer is used to perform convolutional calculations on the calculation result of the previous convolutional layer connected to the current convolutional layer and the historical data of the next layer. The input dataset includes original input data and input layer supplementary data. The input layer supplementary data includes first-side supplementary data and second-side supplementary data. The first-side supplementary data is the supplementary data corresponding to the first digit of the original input data in the input dataset, and the second-side supplementary data is the supplementary data corresponding to the last digit of the original input data in the input dataset. The number of supplementary data on the first side is determined based on the convolution kernel dimension, and the number of supplementary data on the second side is determined based on the number of convolutional layers and is used to supplement the calculation results; The first convolutional layer is used to perform convolution operations with each data in the first set as the first layer calculation benchmark, with x data before each first layer calculation benchmark as historical data, and combining each first layer calculation benchmark and the historical data corresponding to the first layer calculation benchmark. The operation result is the first layer calculation result. The x is the number of supplementary data on the first side. The first set is a data set composed of the first data of the original input data to the last data of the input dataset. The first convolutional layer to the Nth convolutional layer are connected sequentially; The Nth convolutional layer is used to perform convolution operations by taking each data in the Nth set as the sub-layer calculation benchmark, taking the x data before each calculation benchmark as the sub-layer historical data, and combining each sub-layer calculation benchmark and the historical data corresponding to that sub-layer calculation benchmark. The result of the operation is the Nth layer calculation result. The Nth set is a data set composed of the first data of the N-1th layer calculation result to the last data of the input dataset.

2. The structure according to claim 1, characterized in that, The computational structure further includes at least one pooling layer, where N is an even number not less than 2, and the layers are connected sequentially as input layer, each convolutional layer, pooling layer and output layer. The pooling layer is used to pool the computation result of the Nth convolutional layer for the final result output.

3. The structure according to claim 2, characterized in that, The final result is obtained by merging several data points from the Nth layer calculation result of the pooling layer. The Nth layer calculation result includes dirty data and valid data. The dirty data is calculated using x supplementary data as historical data. The Nth layer calculation result is obtained by the Nth convolutional layer performing convolution calculation based on the N-1th layer calculation result and the sub-layer supplementary data of the N-1th layer. The number of dirty data points is determined according to the number of convolutional layers N.

4. A data processing method based on a convolutional neural network computational structure, characterized in that, Applied to real-time streaming data processing scenarios, including speech recognition, the method includes: The first convolutional layer of the neural network computation structure performs convolution calculations on the input dataset of the input layer. The computational convolutional layer performs convolution calculations on the computation results of the previous convolutional layer connected to the current convolutional layer and the historical supplementary data of the next layer. The input dataset includes original input data and input layer supplementary data. The input layer supplementary data includes first-side supplementary data and second-side supplementary data. The first-side supplementary data is the supplementary data corresponding to the first digit of the original input data in the input dataset, and the second-side supplementary data is the supplementary data corresponding to the last digit of the original input data in the input dataset. The number of supplementary data on the first side is determined based on the convolution kernel dimension, and the number of supplementary data on the second side is determined based on the number of convolutional layers and is used to supplement the calculation results; The convolution calculation performed by the computational convolutional layer on the computation result of the previous convolutional layer connected to the current convolutional layer and the historical supplementary data of the next layer includes: The data in the first set are used as the basis for the first-level calculation. Obtain x data points before the calculation baseline of each first layer as historical data; Convolution operation is performed by combining each first-layer calculation benchmark and the historical data corresponding to the first-layer calculation benchmark, and the operation result is determined as the first-layer calculation result. x is the number of supplementary data on the first side, and the first set is a data set composed of the first data of the original input data to the last data of the input dataset. The process of performing convolution calculations by the computational convolutional layer on the computation result of the previous convolutional layer connected to the current convolutional layer and the historical supplementary data of the next layer specifically includes: Obtain each data point in the Nth set as the basis for the next-level calculation; Obtain x data points prior to each calculation baseline as historical data; Convolution operations are performed by combining the computation benchmarks of each sub-layer with the corresponding historical data of the sub-layer, and the result of the operation is determined as the computation result of the Nth layer. The Nth set is a data set composed of the first data of the computation result of the N-1th layer to the last data of the input dataset. The first convolutional layer to the Nth convolutional layer are connected sequentially.

Images