Data processing method and data processing device

Abstract

The present disclosure provides a data processing method and a data processing device, which can be applied to the field of artificial intelligence technology. The method includes: obtaining a data matrix of an image to be processed and a weight matrix of a target convolutional layer from a first cache; dividing the data matrix and the weight matrix into multiple data streams by row, and writing them into multiple second caches respectively, wherein each second cache has a corresponding computing unit; for each second cache, when the written data stream in the second cache meets the computing window requirements, based on a preset step size, the written data stream is constructed into a first data sequence and a second data sequence; based on a preset step size, the first data sequence and the second data sequence are respectively input from both sides of the computing unit, wherein the computing unit includes a plurality of computing nodes connected in series; and the partial products accumulated by the plurality of computing nodes are input into an addition tree to obtain the output value of the current computing window.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to the field of neural network accelerated computing, and more specifically to a data processing method and a data processing device. Background Art

[0002] At present, computing in edge devices such as autonomous driving requires high computing accuracy and low computing latency under low resources and power consumption, but the computing power of related technologies cannot meet the needs of practical applications such as binocular depth estimation. The main reason for restricting computing power is the large amount of input data, which cannot meet the amount of data required for calculation. Therefore, a large amount of data transmission needs to be completed during the calculation process, which increases data communication overhead.

[0003] At present, the mismatch between input data rate and computing power is a challenge faced by dedicated accelerators. For applications such as binocular depth estimation, high real-time computing is required, especially in binocular vision algorithms, which require the simultaneous calculation of two data streams, the left image and the right image, and the two data streams are required to be output at the same time. Edge devices in related technologies are restricted by power consumption and device area, so it is of great significance to calculate two data streams simultaneously with extremely low hardware cost. Summary of the invention

[0004] In view of this, the present disclosure provides a data processing method and a data processing device.

[0005] One aspect of the present disclosure provides a data processing method, comprising:

[0006] Obtaining a data matrix of the image to be processed and a weight matrix of the target convolutional layer from the first buffer;

[0007] Dividing the data matrix and the weight matrix into multiple data streams by row, and writing them into multiple second caches respectively, wherein each of the second caches has a corresponding computing unit;

[0008] For each of the second caches, when the written data stream in the second cache meets the computing window requirement, construct the first data sequence and the second data sequence based on a preset step size.

[0009] Based on the preset step length, the first data sequence and the second data sequence are input from both sides of the computing unit, respectively, wherein the computing unit includes a plurality of computing nodes connected in series; and

[0010] The partial products accumulated by the above calculation nodes are input into the addition tree to obtain the output value of the current calculation window.

[0011] According to an embodiment of the present disclosure, the dimension of the data matrix is ​​M, and the dimension of the weight matrix is ​​N, wherein M is a positive integer, and N is a positive integer less than M;

[0012] The data matrix and the weight matrix are divided into multiple data streams by row, including:

[0013] Splitting the data matrix and the weight matrix by rows to obtain M data vectors and N weight vectors; and

[0014] The (kN+i)th data vector among the M data vectors and the i-th weight vector among the N weight vectors are constructed as the i-th data stream, wherein the k is a positive integer less than 0 or M / N, and the i is a positive integer less than or equal to N.

[0015] According to an embodiment of the present disclosure, the data processing method further includes:

[0016] In a case where the written data stream includes N activation values ​​of the data vector and N weight values ​​of the weight vector, it is determined that the written data stream meets the computation window requirement.

[0017] According to an embodiment of the present disclosure, the step of constructing the first data sequence and the second data sequence from the written data stream based on a preset step size includes:

[0018] Constructing a first data sequence based on the N activation values ​​written into the data stream at intervals of the preset step length; and

[0019] The second data sequence is constructed based on the N weight values ​​written into the data stream at intervals of the preset step size.

[0020] According to an embodiment of the present disclosure,

[0021] The data matrix includes a first data matrix belonging to the first image to be processed and a second data matrix belonging to the second image to be processed;

[0022] The weight matrix includes a first weight matrix belonging to the first target convolution layer and a second weight matrix belonging to the second target convolution layer;

[0023] Among them, the dimension of the above-mentioned first data matrix is ​​equal to the dimension of the second data matrix, and the dimension of the above-mentioned first weight matrix is ​​equal to the dimension of the second weight matrix.

[0024] According to an embodiment of the present disclosure, the step of dividing the data matrix and the weight matrix into multiple data streams by row includes:

[0025] Dividing the first data matrix and the first weight matrix into a plurality of first data streams by row;

[0026] The second data matrix and the second weight matrix are divided into a plurality of second data streams by row.

[0027] According to an embodiment of the present disclosure, the written data stream includes a plurality of activation values ​​and a plurality of weight values ​​in the first data stream, and a plurality of activation values ​​and a plurality of weight values ​​in the second data stream;

[0028] The step of constructing the first data sequence and the second data sequence from the written data stream based on the preset step size includes:

[0029] In the case where the preset step length is 1, the first data sequence is obtained by alternately constructing a plurality of activation values ​​in the first data stream and a plurality of activation values ​​in the second data stream; and

[0030] The second data sequence is obtained by alternately constructing the multiple weight values ​​in the first data stream and the multiple weight values ​​in the second data stream.

[0031] According to an embodiment of the present disclosure, the step of constructing the first data sequence and the second data sequence from the written data stream based on a preset step size includes:

[0032] In the case where the preset step length is n, the first data sequence is obtained by alternately constructing the plurality of activation values ​​in the first data stream and the plurality of activation values ​​in the second data stream at intervals of (n-1) bits, wherein n is a positive integer greater than 1; and

[0033] The second data sequence is obtained by alternately constructing the multiple weight values ​​in the first data stream and the multiple weight values ​​in the second data stream with (n-1) bits as intervals.

[0034] According to an embodiment of the present disclosure, the data processing method further includes:

[0035] The output value of the current calculation window is taken as an element, and the output values ​​of all calculation windows are traversed to obtain an output matrix representing the operation result of the target convolution layer on the data matrix.

[0036] Another aspect of the present disclosure provides a data processing device, comprising:

[0037] An acquisition module, used for acquiring a data matrix of the image to be processed and a weight matrix of a target convolutional layer from the first cache;

[0038] A segmentation module, used for segmenting the data matrix and the weight matrix into a plurality of data streams by row, and writing the data streams into a plurality of second caches respectively, wherein each of the second caches has a corresponding computing unit;

[0039] A data sequence construction module, configured to construct, for each of the second caches, a first data sequence and a second data sequence based on a preset step length, the written data stream in the second cache when the written data stream meets the computing window requirement;

[0040] a calculation module, configured to input the first data sequence and the second data sequence from both sides of the calculation unit based on the preset step size, wherein the calculation unit includes a plurality of calculation nodes connected in series; and

[0041] The output module is used to input the partial products accumulated by the plurality of the above-mentioned computing nodes into the addition tree to obtain the output value of the current computing window.

[0042] Based on the above technical solutions, the data processing method and data processing device disclosed in the present invention have at least one of the following advantages over the prior art:

[0043] The data processing method provided by the present disclosure can operate on the left and right disparity maps collected by the binocular camera. Through the convolution calculation of the left and right disparity maps, it can solve the problem of deploying the stereo matching algorithm in binocular vision in edge devices such as FPGA, reduce the hardware resources required for calculation, reduce the waiting for dual data streams in hardware execution, and improve the practicality and real-time performance of algorithm deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0045] Figure 1 An exemplary system architecture to which the data processing method of the present disclosure can be applied is schematically shown;

[0046] Figure 2 A flowchart schematically shows a data processing method according to an embodiment of the present disclosure.

[0047] Figure 3 A schematic diagram schematically shows a computing node according to an embodiment of the present disclosure;

[0048] Figure 4 A schematic diagram of a computing unit according to an embodiment of the present disclosure is schematically shown;

[0049] Figure 5 A schematic diagram of a computing array according to an embodiment of the present disclosure is schematically shown;

[0050] Figure 6 The structure of a computing unit according to an embodiment of the present disclosure is schematically shown;

[0051] Figure 7 A schematic diagram schematically shows the calculation results of each time node of a systolic array computing dual data stream task according to an embodiment of the present disclosure;

[0052] Figure 8 A block diagram schematically shows a data processing device according to an embodiment of the present disclosure;

[0053] Fig. 9 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0054] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0055] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0056] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0057] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0058] In the technical solution disclosed in the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0059] In the process of realizing the inventive concept of the present disclosure, the inventors found that using hardware to accelerate multiplication and accumulation operations is a key acceleration direction for deep learning accelerators. For example, accelerating a single CNN (convolutional neural network) layer can play a decisive role in improving the calculation speed. Traditional systolic arrays can be applied to CNN window calculations to accelerate the calculation process of convolutional neural networks. Traditional systolic arrays can fix weights in the systolic array, flow parameters, and finally use addition trees to complete an output. This solution does not consider the flow of weights and has low flexibility.

[0060] Specifically, an embodiment of the present disclosure provides a data processing method and a data processing device. The method includes: obtaining a data matrix of an image to be processed and a weight matrix of a target convolutional layer from a first cache; dividing the data matrix and the weight matrix into multiple data streams by row, and writing them into multiple second caches respectively, wherein each second cache has a corresponding computing unit; for each second cache, when the written data stream in the second cache meets the computing window requirements, based on a preset step size, constructing a first data sequence and a second data sequence from the written data stream; based on a preset step size, inputting the first data sequence and the second data sequence from both sides of the computing unit respectively, wherein the computing unit includes multiple computing nodes connected in series; and inputting the partial products accumulated by multiple computing nodes into an addition tree to obtain the output value of the current computing window.

[0061] According to the embodiments of the present disclosure, the data processing method can compress the model scale from the software level, reduce the storage space of the model and the amount of data transmission, and can also use the present disclosure to calculate the compressed data. In summary, the data processing method of the present disclosure can calculate two data streams at the same time with extremely low hardware consumption, and the output results are output alternately in the same array, ensuring the timing synchronization of the two data streams.

[0062] In the technical solution disclosed in the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0063] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0064] Figure 1 An exemplary system architecture to which the data processing method of the present disclosure can be applied is schematically shown.

[0065] like Figure 1 As shown, the data matrix of the image to be processed and the weight matrix of the target convolution layer can be stored in the first cache 110, and the data matrix and the weight matrix are divided into multiple data streams by row, and are written into the second caches 121, 122, 123, 124, 125, and 126 respectively, wherein each second cache has a corresponding calculation unit; for each second cache, when the written data stream in the second cache meets the calculation window requirements, based on the preset step length, the written data stream is constructed into a first data sequence and a second data sequence; based on the preset step length, the first data sequence and the second data sequence are input from both sides of the calculation unit respectively, wherein the calculation unit includes a plurality of calculation nodes connected in series; and the partial products accumulated by the plurality of calculation nodes are input into the addition tree to obtain the output value of the current calculation window. The first calculation storage module 131 can store the data required for the calculation of the second caches 121, 122, and 123, and the second calculation storage module can store the data required for the calculation of 124, 125, and 126, and the output value of the calculation window can be transmitted to the output value storage module 140.

[0066] Figure 2 The flowchart of the data processing method according to the embodiment of the present disclosure is schematically shown.

[0067] like Figure 2 As shown, the data processing method includes operations S201 to S205.

[0068] In operation S201, a data matrix of an image to be processed and a weight matrix of a target convolutional layer are obtained from a first buffer.

[0069] In operation S202, the data matrix and the weight matrix are divided into a plurality of data streams by row, and are written into a plurality of second caches respectively, wherein each second cache has a corresponding computing unit.

[0070] In operation S203, for each second cache, when the written data stream in the second cache meets the computing window requirement, the written data stream is constructed into a first data sequence and a second data sequence based on a preset step size.

[0071] In operation S204, based on a preset step length, the first data sequence and the second data sequence are respectively input from both sides of a computing unit, wherein the computing unit includes a plurality of computing nodes connected in series.

[0072] In operation S205, partial products accumulated by a plurality of computing nodes are input into an addition tree to obtain an output value of the current computing window.

[0073] According to an embodiment of the present disclosure, the dimension of the data matrix is ​​M, and the dimension of the weight matrix is ​​N, wherein M is a positive integer and N is a positive integer less than M;

[0074] Among them, the data matrix and the weight matrix are divided into multiple data streams by row, including:

[0075] Split the data matrix and the weight matrix by rows to obtain M data vectors and N weight vectors; and

[0076] The (kN+i)th data vector among the M data vectors and the i-th weight vector among the N weight vectors are constructed into the i-th data stream, where k is 0 or a positive integer less than M / N, and i is a positive integer less than or equal to N.

[0077] According to an embodiment of the present disclosure, the data processing method further includes:

[0078] In a case where the written data stream includes N activation values ​​of the data vector and N weight values ​​of the weight vector, it is determined that the written data stream meets the computation window requirement.

[0079] According to an embodiment of the present disclosure, the step of constructing a first data sequence and a second data sequence from a written data stream based on a preset step size includes:

[0080] constructing a first data sequence based on the N activation values ​​written into the data stream at intervals of a preset step length; and

[0081] A second data sequence is constructed based on the N weight values ​​written into the data stream at intervals of a preset step length.

[0082] According to an embodiment of the present disclosure,

[0083] The data matrix includes a first data matrix belonging to the first image to be processed and a second data matrix belonging to the second image to be processed;

[0084] The weight matrix includes a first weight matrix belonging to the first target convolution layer and a second weight matrix belonging to the second target convolution layer;

[0085] The dimension of the first data matrix is ​​equal to the dimension of the second data matrix, and the dimension of the first weight matrix is ​​equal to the dimension of the second weight matrix.

[0086] According to an embodiment of the present disclosure, the data matrix and the weight matrix are divided into multiple data streams by row, including:

[0087] Splitting the first data matrix and the first weight matrix into a plurality of first data streams by row;

[0088] The second data matrix and the second weight matrix are divided into a plurality of second data streams by row.

[0089] According to an embodiment of the present disclosure, wherein the written data stream includes a plurality of activation values ​​and a plurality of weight values ​​in the first data stream, a plurality of activation values ​​and a plurality of weight values ​​in the second data stream;

[0090] The step of constructing a first data sequence and a second data sequence from the written data stream based on a preset step size includes:

[0091] When the preset step size is 1, a plurality of activation values ​​in the first data stream and a plurality of activation values ​​in the second data stream are alternately constructed to obtain a first data sequence; and

[0092] The second data sequence is obtained by alternately constructing a plurality of weight values ​​in the first data stream and a plurality of weight values ​​in the second data stream.

[0093] According to an embodiment of the present disclosure, the step of constructing a first data sequence and a second data sequence from a written data stream based on a preset step size includes:

[0094] In the case where the preset step length is n, multiple activation values ​​in the first data stream and multiple activation values ​​in the second data stream are alternately constructed at intervals of (n-1) bits to obtain a first data sequence, wherein n is a positive integer greater than 1; and

[0095] The multiple weight values ​​in the first data stream and the multiple weight values ​​in the second data stream are alternately constructed with (n-1) bits as intervals to obtain a second data sequence.

[0096] According to an embodiment of the present disclosure, the data processing method further includes:

[0097] The output value of the current calculation window is taken as an element, and the output values ​​of all calculation windows are traversed to obtain an output matrix used to represent the operation result of the target convolution layer on the data matrix.

[0098] In one embodiment of the present disclosure, a data processing method is provided, including:

[0099] Operation S1, obtaining a data matrix of an image to be processed and a weight matrix of a target convolutional layer from a first cache.

[0100] According to an embodiment of the present disclosure, the data matrix and the weight matrix of the target convolutional layer can be transmitted to the first cache through the AXI bus, but is not limited to this. The data collected by the camera in real time can also be sent to the first cache.

[0101] Operation S2, dividing the data matrix and the weight matrix into multiple data streams by row, and writing them into multiple second caches respectively, wherein each second cache has a corresponding computing unit.

[0102] According to an embodiment of the present disclosure, the calculation data of each calculation unit needs to include at least the data required for a complete multiplication and accumulation calculation, wherein the multiplication and accumulation calculation can be a convolution calculation. Then, the data is sent to the second cache in batches. In the second cache, an indication signal is set to indicate that the single data amount meets the minimum calculation requirement. The calculation array control circuit starts a complete calculation according to the indication signal, and the data continues to be sent to the array from the first cache. When the data is insufficient to complete a complete window calculation, the calculation is stopped.

[0103] According to an embodiment of the present disclosure, the data stream may include grayscale image data, etc.

[0104] According to an embodiment of the present disclosure, a computing array may include multiple computing units, and each computing unit may include multiple computing nodes.

[0105] Operation S3: for each second cache, when the written data stream in the second cache meets the computing window requirement, construct the first data sequence and the second data sequence based on a preset step size for the written data stream.

[0106] Operation S4: based on a preset step length, input the first data sequence and the second data sequence from two sides of a computing unit respectively, wherein the computing unit includes a plurality of computing nodes connected in series.

[0107] Figure 3 A schematic diagram of a computing node according to an embodiment of the present disclosure is schematically shown.

[0108] like Figure 3 As shown, the computing node 310 is a multiplication operation.

[0109] It can be expressed by formula (4).

[0110] y=w×x; (4)

[0111] In formula (4), w is a weight vector of the data stream, and x is an activation vector of the data stream. After the calculation is completed, it can be stored in the dedicated cache 321 and 322 of the computing unit. The result can be expressed as:

[0112] y 1 =y 1 +y

[0113] y 2 =y 2 +y

[0114] where y 1 Represents the dedicated cache result of data stream 1, y 2 Represents the dedicated buffer result of data stream 2.

[0115] Figure 4 A schematic diagram of a computing unit according to an embodiment of the present disclosure is schematically shown.

[0116] like Figure 4 As shown, the computing unit 400 may include three computing nodes 411, 412, and 413 connected in series. The first data sequence may include activation values ​​Rx1, Lx1, Rx2, and Lx2, and the second data sequence may also include weight values ​​Rw1, Lw1, Rw2, and Lw2. The first data sequence and the second data sequence are input from both sides of the computing unit 400 respectively. The computing unit 400 can perform multiplication and addition calculations on the bidirectionally flowing first data sequence Rx1, Lx1, Rx2, Lx2 and the second data sequence Rw1, Lw1, Rw2, and Lw2 in a pulsating form, and alternate the two data sequences. The data sequence enters the computing array in such a way that the activation value in the data stream is sent into the array in a forward direction according to the beat, and the weight value in the data stream is sent into the computing array in a reverse direction according to the beat.

[0117] Figure 5 A schematic diagram of a computing array according to an embodiment of the present disclosure is schematically shown.

[0118] like Figure 5 As shown, the computing array 500 may include three computing units. The computing array 500 may be a bidirectional dual-data stream systolic computing array applied to a deep learning accelerator, and the computing content may be represented by formula (5).

[0119]

[0120] In formula (5), datas is the total data block, data is the calculation window data of the input data stream, len() is the data amount, y[k] is the calculation result of the kth window, w is the weight matrix, and x is the data matrix. Figure 5 The three computing nodes in the vertical direction form an addition tree. The two-way data stream systolic array structure is used to calculate the two-way data stream A single computing unit can obtain the calculation result after the calculation is completed, including the partial product of the partial calculation window of the first data stream. N computing units can obtain the partial product of the complete calculation window of the first data stream, N is the number of rows of the window size, and N partial products are accumulated in the dedicated cache of the first data stream in the computing unit in each odd cycle, and N calculation partial accumulation results can be obtained. Finally, all the accumulated partial products are sent to the addition tree to obtain the output value of the first data stream in a calculation window. For the calculation of the second data stream, the weight value and activation value of the current calculation window of the second data stream are sent in all even cycles, one group at a time, and these weight values ​​and activation values ​​will flow in opposite directions in the calculation array, with another group of data streams in between. N computing units can obtain the partial product of the complete calculation window of the second data stream, N is the number of rows of the window size, and N partial products are accumulated in the dedicated cache of the second data stream in the computing unit in each even cycle, and the accumulated results of N calculation partial products can be obtained. Finally, all the accumulated results of all these partial products are sent to the addition tree to obtain the output value of the final calculation window of the second data stream.

[0121] Figure 6 The structure of a computing unit according to an embodiment of the present disclosure is schematically shown.

[0122] like Figure 6 As shown, the dedicated caches 611 and 612 can be used to store the first data stream and the second data stream, which are sent to the serial accumulation unit after the odd-even cycle judgment by the judgment unit 621, and added with the result from the multiplication operation, and then stored in the dedicated cache after being accumulated with the existing value in the dedicated cache, replacing the original value, and completing the update of the value in the dedicated cache.

[0123] After obtaining the output value of the entire calculation window, the results of all parallel calculation units in the calculation array are sent to the addition tree, the value in the dedicated cache is reset to 0, and the accumulation of the partial products of the corresponding data streams begins again; the addition tree performs a summation operation based on the input values ​​in all corresponding dedicated caches to obtain the calculation result of the complete calculation window, and stores the calculation result in the output dedicated cache; based on the calculation array of the bidirectional dual data stream and the calculation process when its data stream flows, the partial product results in the calculation window of the first data stream and the second data stream can be obtained, and the partial product results are sent to the addition tree to finally obtain the output value of the calculation window.

[0124] Figure 7 The figure schematically shows the calculation results of each time node according to the embodiment of the present disclosure.

[0125] like Figure 7 As shown, when the preset step size is 1, the activation values ​​rx0, rx1, rx2 in the first data stream and the activation values ​​1x0, lx1, 1x2 in the second data stream are alternately constructed to obtain a first data sequence; the weight values ​​rw0, rw1, rw2 in the first data stream and the multiple weight values ​​lw0, lw1, 1w2 in the second data stream are alternately constructed to obtain a second data sequence.

[0126] A 3*3 computing array can be composed of three 3*1 computing units. In a 3*1 computing unit, each computing node can calculate the first data flow in such a way that x flows from computing node 0 to computing node 2, and w flows from computing node 2 to computing node 0, and the reusable weight is calculated. When w flows to computing node 0, it returns to computing node 2.

[0127] At time 0: rx0 is on compute node 0, rw0 is on compute node 2;

[0128] At time 1: rx0 is on computing node 1, rw0 is on computing node 1; lx0 is on computing node 0, lw0 is on computing node 2;

[0129] At time 2: rx0 is on computing node 2, rw0 is on computing node 0; lx0 is on computing node 1, lw0 is on computing node 1; rx1 is on computing node 0, rw1 is on computing node 2;

[0130] At time 3: lw1 is on compute node 2; lx0 is on compute node 2, lw0 is on compute node 0; rx1 is on compute node 1, rw1 is on compute node 1; lx1 is on compute node 0;

[0131] At time 4: rx1 is on compute node 2; lw1 is on compute node 2; lx1 is on compute node 1, rw1 is on compute node 1; rx2 is on compute node 0, lw0 is on compute node 0.

[0132] The activation values ​​and weight values ​​of the first data stream and the second data stream flow into the computing unit from opposite sides respectively. The weight values ​​and activation values ​​of the first data stream and the second data stream are alternately input into the computing unit; the computing status of computing node 1 can be represented by Table 1.

[0133] Table 1

[0134] Moment 1 Compute node 1 outputs rx0*rw0 Moment 2 Compute node 1 outputs lx0*lw0 Moment 3 Compute node 1 outputs rx1*rw1 Moment 4 Compute node 1 outputs lx1*lw1

[0135] In computing node 1, the products of the first data stream in the first window are output at time 1, time 3, and time 5 respectively, and these products are accumulated and calculated in the first data stream dedicated cache according to the rule of formula (6).

[0136] y 1 =y 1 +y(odd operation); (6)

[0137] N parallel paths synchronously complete N calculation processes and send them to the addition tree to obtain the calculation results of the complete window.

[0138] In computing node 1, the products of the first window of the second data stream are output at time 2, time 4, and time 6 respectively, and these products are accumulated and calculated in the dedicated cache of the second data stream according to the rule of formula (7).

[0139] y 2 =y 2 +y(even operation) (7)

[0140] N parallel paths synchronously complete N of the calculation processes and send them to the same addition tree as the first data stream. The calculation results of the complete window are obtained by multiplexing the addition tree in different clock cycles.

[0141] Figure 8 The block diagram schematically shows a data processing device according to an embodiment of the present disclosure.

[0142] like Figure 8 As shown, the data processing device 800 according to the embodiment of the present disclosure may include: an acquisition module 810, a segmentation module 820, a data sequence construction module 830, a calculation module 840 and an output module 850.

[0143] An acquisition module 810 is used to acquire a data matrix of the image to be processed and a weight matrix of a target convolutional layer from a first buffer;

[0144] A segmentation module 820 is used to segment the data matrix and the weight matrix into multiple data streams by row, and write them into multiple second caches respectively, wherein each second cache has a corresponding computing unit;

[0145] A data sequence construction module 830 is used to construct a first data sequence and a second data sequence from the written data stream based on a preset step length for each second cache when the written data stream in the second cache meets the computing window requirement;

[0146] A calculation module 840, configured to input the first data sequence and the second data sequence from both sides of a calculation unit based on a preset step length, wherein the calculation unit includes a plurality of calculation nodes connected in series; and

[0147] The output module 850 is used to input the partial products accumulated by multiple computing nodes into the addition tree to obtain the output value of the current computing window.

[0148] According to an embodiment of the present disclosure, the dimension of the data matrix is ​​M, and the dimension of the weight matrix is ​​N, wherein M is a positive integer and N is a positive integer less than M;

[0149] The segmentation module may include:

[0150] The separation unit is used to split the data matrix and the weight matrix by row to obtain M data vectors and N weight vectors.

[0151] A construction unit is used to construct the (kN+i)th data vector among the M data vectors and the i-th weight vector among the N weight vectors into the i-th data stream, wherein k is 0 or a positive integer less than M / N, and i is a positive integer less than or equal to N.

[0152] According to an embodiment of the present disclosure, the above-mentioned data processing device may further include: a determination module.

[0153] A determination module is used to determine whether the written data stream meets the calculation window requirement when the written data stream includes N activation values ​​of the data vector and N weight values ​​of the weight vector.

[0154] According to an embodiment of the present disclosure, the data sequence construction module may include: a first data sequence construction unit and a second data sequence construction unit.

[0155] The first data sequence construction unit is used to construct a first data sequence based on N activation values ​​written into the data stream at intervals of a preset step length.

[0156] The second data sequence construction unit is used to construct a second data sequence based on the N weight values ​​written into the data stream at intervals of a preset step length.

[0157] According to an embodiment of the present disclosure,

[0158] The data matrix includes a first data matrix belonging to the first image to be processed and a second data matrix belonging to the second image to be processed;

[0159] The weight matrix includes a first weight matrix belonging to the first target convolution layer and a second weight matrix belonging to the second target convolution layer;

[0160] The dimension of the first data matrix is ​​equal to the dimension of the second data matrix, and the dimension of the first weight matrix is ​​equal to the dimension of the second weight matrix.

[0161] According to an embodiment of the present disclosure, the partition module may include: a first partition unit and a second partition unit.

[0162] The first separation unit is used to separate the first data matrix and the first weight matrix into a plurality of first data streams by row.

[0163] The second separation unit is used to divide the second data matrix and the second weight matrix into multiple second data streams by row.

[0164] According to an embodiment of the present disclosure, the written data stream includes multiple activation values ​​and multiple weight values ​​in the first data stream, and multiple activation values ​​and multiple weight values ​​in the second data stream.

[0165] The data sequence construction module may include: a first alternating construction unit and a second alternating construction unit.

[0166] A first alternating construction unit is used to alternately construct a plurality of activation values ​​in the first data stream and a plurality of activation values ​​in the second data stream to obtain a first data sequence when a preset step length is 1; and

[0167] The second alternating construction unit is used to alternately construct multiple weight values ​​in the first data stream and multiple weight values ​​in the second data stream to obtain a second data sequence.

[0168] According to an embodiment of the present disclosure, the data sequence construction module may include a third alternating construction unit and a fourth alternating construction unit.

[0169] A third alternating construction unit is configured to, when a preset step length is n, alternately construct a plurality of activation values ​​in the first data stream and a plurality of activation values ​​in the second data stream at intervals of (n-1) bits to obtain a first data sequence, wherein n is a positive integer greater than 1; and

[0170] The fourth alternating construction unit is used to alternately construct multiple weight values ​​in the first data stream and multiple weight values ​​in the second data stream at intervals of (n-1) bits to obtain a second data sequence.

[0171] According to an embodiment of the present disclosure, the data processing device may further include a traversal module.

[0172] The traversal module is used to take the output value of the current calculation window as an element, traverse the output values ​​of all calculation windows, and obtain an output matrix used to represent the operation result of the target convolution layer on the data matrix.

[0173] According to the modules and units of the embodiments of the present disclosure, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules and units of the embodiments of the present disclosure, any one or more can be split into multiple modules for implementation. According to the modules and units of the embodiments of the present disclosure, any one or more can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, according to the modules and units of the embodiments of the present disclosure, one or more can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0174] For example, any multiple of the acquisition module 810, the segmentation module 820, the data sequence construction module 830, the calculation module 840 and the output module 850 can be combined in one module / unit / subunit for implementation, or any one of the modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functions of one or more modules / units / subunits in these modules / units / subunits can be combined with at least part of the functions of other modules / units / subunits and implemented in one module / unit / subunit. According to an embodiment of the present disclosure, at least one of the acquisition module 810, the segmentation module 820, the data sequence construction module 830, the calculation module 840 and the output module 850 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a suitable combination of any of them. Alternatively, at least one of the acquisition module 810, the segmentation module 820, the data sequence construction module 830, the calculation module 840 and the output module 850 can be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function can be performed.

[0175] It should be noted that the data processing device part in the embodiments of the present disclosure corresponds to the data processing method part in the embodiments of the present disclosure. The description of the data processing device part specifically refers to the data processing method part, which will not be repeated here.

[0176] Fig. 9A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure is schematically shown. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0177] like Fig. 9 As shown, the computer electronic device 900 according to the embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 902 or the program loaded from the storage part 908 to the random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present disclosure.

[0178] In RAM 903, various programs and data required for the operation of electronic device 900 are stored. Processor 901, ROM 902 and RAM 903 are connected to each other via bus 904. Processor 901 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 902 and / or RAM 903. It should be noted that the program can also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0179] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the I / O interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0180] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0181] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0182] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0183] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 902 and / or the RAM 903 described above and / or one or more memories other than the ROM 902 and the RAM 903 .

[0184] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains a program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the XX method provided by the embodiment of the present disclosure.

[0185] When the computer program is executed by the processor 901, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0186] In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and downloaded and installed through the communication part 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0187] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0188] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0189] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data processing method, comprising: Obtaining a data matrix of the image to be processed and a weight matrix of the target convolutional layer from the first buffer; Dividing the data matrix and the weight matrix into a plurality of data streams by row, and writing the data streams into a plurality of second caches respectively, wherein each of the second caches has a corresponding computing unit; For each of the second caches, when the written data stream in the second cache meets the computing window requirement, construct the first data sequence and the second data sequence from the written data stream based on a preset step size; Based on the preset step length, inputting the first data sequence and the second data sequence from two sides of the computing unit respectively, wherein the computing unit includes a plurality of computing nodes connected in series; and Inputting the partial products accumulated by the plurality of computing nodes into an addition tree to obtain an output value of the current computing window; Wherein, the dimension of the data matrix is ​​M, and the dimension of the weight matrix is ​​N, wherein M is a positive integer, and N is a positive integer less than M; Wherein, dividing the data matrix and the weight matrix into multiple data streams by row includes: Splitting the data matrix and the weight matrix by rows to obtain M data vectors and N weight vectors; and The (kN+i)th data vector among the M data vectors and the i-th weight vector among the N weight vectors are constructed into the i-th data stream, wherein k is 0 or a positive integer less than M / N, and i is a positive integer less than or equal to N.

2. The method according to claim 1, further comprising: In a case where the written data stream includes N activation values ​​of the data vector and N weight values ​​of the weight vector, it is determined that the written data stream meets the computation window requirement.

3. The method according to claim 2, wherein: The step of constructing the first data sequence and the second data sequence from the written data stream based on the preset step length includes: constructing a first data sequence based on the N activation values ​​in the written data stream at intervals of the preset step size; and A second data sequence is constructed based on the N weight values ​​written into the data stream at intervals of the preset step length.

4. The method according to claim 1, wherein: The data matrix includes a first data matrix belonging to the first image to be processed and a second data matrix belonging to the second image to be processed; The weight matrix includes a first weight matrix belonging to the first target convolution layer and a second weight matrix belonging to the second target convolution layer; The dimension of the first data matrix is ​​equal to the dimension of the second data matrix, and the dimension of the first weight matrix is ​​equal to the dimension of the second weight matrix.

5. The method according to claim 4, wherein: The step of dividing the data matrix and the weight matrix into a plurality of data streams by row comprises: Splitting the first data matrix and the first weight matrix into a plurality of first data streams by row; The second data matrix and the second weight matrix are divided into a plurality of second data streams by row.

6. The method according to claim 5, wherein: The written data stream includes a plurality of activation values ​​and a plurality of weight values ​​in the first data stream, and a plurality of activation values ​​and a plurality of weight values ​​in the second data stream; The step of constructing the first data sequence and the second data sequence from the written data stream based on the preset step size includes: When the preset step size is 1, the first data sequence is obtained by alternately constructing a plurality of activation values ​​in the first data stream and a plurality of activation values ​​in the second data stream; and The second data sequence is obtained by alternately constructing multiple weight values ​​in the first data stream and multiple weight values ​​in the second data stream.

7. The method according to claim 6, wherein: The step of constructing the first data sequence and the second data sequence from the written data stream based on the preset step length includes: When the preset step length is n, multiple activation values ​​in the first data stream and multiple activation values ​​in the second data stream are alternately constructed with (n-1) bits as intervals to obtain the first data sequence, wherein n is a positive integer greater than 1; and The second data sequence is obtained by alternately constructing multiple weight values ​​in the first data stream and multiple weight values ​​in the second data stream at intervals of (n-1) bits.

8. The method according to claim 1, further comprising: The output value of the current calculation window is taken as an element, and the output values ​​of all calculation windows are traversed to obtain an output matrix for representing the operation result of the target convolution layer on the data matrix.

9. A data processing device, comprising: An acquisition module, used for acquiring a data matrix of the image to be processed and a weight matrix of a target convolutional layer from the first cache; A segmentation module, used for segmenting the data matrix and the weight matrix into multiple data streams by row, and writing the data streams into multiple second caches respectively, wherein each second cache has a corresponding computing unit; A data sequence construction module, configured to construct, for each second cache, a first data sequence and a second data sequence based on a preset step length, the written data stream in the second cache when the written data stream meets the computing window requirement; a calculation module, configured to input the first data sequence and the second data sequence from both sides of the calculation unit respectively based on the preset step size, wherein the calculation unit includes a plurality of calculation nodes connected in series; and An output module, used for inputting the partial products accumulated by the plurality of computing nodes into an addition tree to obtain an output value of a current computing window; Wherein, the dimension of the data matrix is ​​M, and the dimension of the weight matrix is ​​N, wherein M is a positive integer, and N is a positive integer less than M; The segmentation module includes: a separation unit, used for dividing the data matrix and the weight matrix by row to obtain M data vectors and N weight vectors; and A construction unit, used to construct the (kN+i)th data vector among the M data vectors and the i-th weight vector among the N weight vectors into the i-th data stream, wherein the k is 0 or a positive integer less than M / N, and the i is a positive integer less than or equal to N.

Images