Deep neural network acceleration method and system using ring tensor

By using the directional nonlinear activation function in the multiplication operation step of the ring tensor, the problem of increasing the number of multipliers in the prior art is solved, and efficient channel information exchange and image quality maintenance are achieved.

CN113780538BActive Publication Date: 2024-11-12黄朝宗
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202110134690.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-06-09
Filing Date
2021-02-01
Publication Date
2024-11-12
Estimated Expiration
2041-02-01

AI Technical Summary

Technical Problem

When using ring tensors to accelerate deep neural networks, the number of multiplier bits is added to exchange inter-channel information, resulting in increased complexity and difficulty in maintaining image quality.

Method used

A nonlinear activation function with direction is used to place it in the multiplication operation step of the ring tensor. Through the multiplication and addition of ring elements, the multiplication and addition of multiplication are kept from increasing the number of multiplier bits, and the highest exchange of image quality is achieved in ring algebra.

Benefits of technology

Effectively exchange inter-channel information, avoid increasing the number of multiplier bits, maintain image quality, and achieve the highest image quality in ring algebra.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113780538B_ABST
    Figure CN113780538B_ABST
Patent Text Reader

Abstract

The present invention provides a deep neural network acceleration method using a ring tensor, wherein the ring tensor setting step is to set the input feature ring tensor and parameter ring tensor of the convolutional network, the input feature ring tensor includes multiple input feature ring elements, and the parameter ring tensor includes multiple parameter ring elements. The ring tensor convolution operation step is to operate on the input feature ring elements and the parameter ring elements according to the ring element multiplication operation step and the ring element addition operation step to produce multiple convolution feature ring elements of the convolution feature ring tensor. The tensor nonlinear activation function operation step is to perform a directional nonlinear activation function on one of the convolution feature ring elements to generate an output feature ring element. Therefore, by using a directional nonlinear activation function, the number of bits of the multiplier can be kept constant and the image quality can be maintained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a deep neural network acceleration method and a system thereof, and in particular to a deep neural network acceleration method using a ring tensor and a system thereof. Background Art

[0002] When using convolutional neural networks in image processing applications, the computational requirements are quite high. The mainstream research uses parameter pruning (weight pruning) to remove network parameters with small values ​​or unnecessary values ​​to reduce the amount of computation and the required parameter storage space. However, the computational flow of the pruned network becomes irregular, which in turn causes a burden during hardware acceleration, and is therefore not suitable for accelerators that use massive parallel computing.

[0003] On the other hand, there are some studies exploring how to reduce the amount of computation while retaining the regularity of the network, including Quaternion Network, Circulant Convolutional Neural Network (CirCNN), Shuffle Network (ShuffleNet) and Convolutional Neural Network (HadaNet) using Hadamard Transform. The common point of these approaches is to use the sparsity of feature vectors in the channel direction to regularly reduce the amount of computation and the number of parameters. In order to remedy the quality loss caused by fewer multiplications, it is necessary to allow additional information between different channels to be exchanged. Conceptually, it can be divided into additional linear transformations between channels and channel transformation positions. The linear transformation method can obtain better quality, but for common fixed-point operations, these transformations will increase the number of bits of feature values ​​and parameters, which will lead to an increase in the complexity of the multiplier.

[0004] It can be seen that the market currently lacks a deep neural network acceleration method and system using ring tensors that can effectively exchange information between channels and maintain image quality without increasing the number of bits of the multiplier, so related industry players are all looking for solutions. Summary of the invention

[0005] Therefore, the purpose of the present invention is to provide a deep neural network acceleration method and system using ring tensors, which utilizes a directional nonlinear activation function to maintain image quality without increasing the number of bits of the multiplier in the ring element multiplication operation step, thereby avoiding the problem of increased complexity caused by increasing the number of bits of the multiplier in the prior art.

[0006] According to one embodiment of the method embodiment of the present invention, a deep neural network acceleration method using a ring tensor is provided, comprising the following steps: a ring tensor setting step, a ring tensor convolution operation step, and a tensor nonlinear activation function operation step. The ring tensor setting step is to set the input feature ring tensor and the parameter ring tensor of the convolution network, the input feature ring tensor includes a plurality of input feature ring elements, and the parameter ring tensor includes a plurality of parameter ring elements. The ring tensor convolution operation step is to operate on these input feature ring elements of the input feature ring tensor and these parameter ring elements of the parameter ring tensor according to the ring element multiplication operation step and the ring element addition operation step to produce a plurality of convolution feature ring elements of the convolution feature ring tensor. The tensor nonlinear activation function operation step is to perform a directional nonlinear activation function on one of these convolution feature ring elements of the convolution feature ring tensor to generate an output feature ring element. The ring element multiplication operation step includes performing ring element multiplication on one of these input feature ring elements and one of these parameter ring elements to generate a multiplication output ring element. The multiplication output components of the multiplication output ring elements are obtained by performing component-wise multiplication of the input feature components of one of the input feature ring elements and the parameter components of one of the parameter ring elements. The ring element addition operation step includes performing ring element addition on a plurality of multiplication output ring elements to generate one of the convolution feature ring elements. The convolution feature components of each convolution feature ring element are obtained by performing component-wise summation of a plurality of multiplication output components of the multiplication output ring elements. The deep neural network acceleration method using the ring tensor is applied to images.

[0007] Therefore, the deep neural network acceleration method using ring tensors of the present invention can place additional linear transformations in the directional nonlinear activation function through the directional nonlinear activation function, so as not to increase the number of bits of the multiplier in the ring element multiplication operation step. In addition, the present invention can effectively exchange information between channels and achieve the highest image quality in the ring algebra.

[0008] Other embodiments of the aforementioned implementation are as follows: The aforementioned directional nonlinear activation function may include a first conversion step, a linear rectification function operation step, and a second conversion step. The first conversion step is to perform a first linear conversion on one of these convolution feature ring elements to generate a linear output ring element. The linear rectification function operation step is to perform a component linear rectification function on the linear output ring element to generate a linear rectification output ring element. The second conversion step is to perform a second linear conversion on the linear rectification output ring element to generate an output feature ring element.

[0009] Other examples of the aforementioned implementation are as follows: any one of the aforementioned first linear transformation and the second linear transformation may be a Hadamard Transform.

[0010] Other embodiments of the aforementioned implementation are as follows: The aforementioned directional nonlinear activation function may include a first conversion step, a normalized linear conversion step, a linear rectification function operation step, and a second conversion step. The first conversion step is to perform a first linear conversion on one of these convolution feature ring elements to generate a first linear output ring element. The normalized linear conversion step is to perform ring element multiplication on the first linear output ring element and the normalized linear conversion to generate a second linear output ring element. The linear rectification function operation step is to perform a component linear rectification function on the second linear output ring element to generate a linear rectification output ring element. The second conversion step is to perform a second linear conversion on the linear rectification output ring element to generate an output feature ring element. Either the first linear conversion or the second linear conversion is a Hadamard conversion.

[0011] Other embodiments of the aforementioned implementation are as follows: The aforementioned deep neural network acceleration method using a ring tensor may further include a tensor quantization step, which performs a quantization operation on the output feature ring element to generate a quantized output ring element. The ring tensor setting step, the ring tensor convolution operation step, the tensor nonlinear activation function operation step and the tensor quantization step are performed in sequence.

[0012] According to one embodiment of the structural implementation of the present invention, a deep neural network acceleration system using a ring tensor is provided, which includes a first memory, a second memory and an operation processing unit. The first memory is used to access the input feature ring tensor of the convolutional network, and the input feature ring tensor includes multiple input feature ring elements. The second memory is used to access the parameter ring tensor of the convolutional network, and the parameter ring tensor includes multiple parameter ring elements. The operation processing unit is electrically connected to the first memory and the second memory, and the operation processing unit receives the input feature ring tensor and the parameter ring tensor and is configured to implement an operation including the following steps: a ring tensor setting step, a ring tensor convolution operation step, and a tensor nonlinear activation function operation step. The ring tensor setting step is to set the input feature ring tensor and the parameter ring tensor of the convolutional network. The ring tensor convolution operation step is to operate on these input feature ring elements of the input feature ring tensor and these parameter ring elements of the parameter ring tensor according to the ring element multiplication operation step and the ring element addition operation step to produce multiple convolution feature ring elements of the convolution feature ring tensor. The tensor nonlinear activation function operation step is to perform a directional nonlinear activation function on one of these convolution feature ring elements of the convolution feature ring tensor to generate an output feature ring element. The ring element multiplication operation step includes performing ring element multiplication on one of these input feature ring elements and one of these parameter ring elements to generate a multiplication output ring element. The multiplication output component of the multiplication output ring element is obtained by performing component product on the input feature components of one of these input feature ring elements and the parameter components of one of these parameter ring elements. The ring element addition operation step includes performing ring element addition on multiple multiplication output ring elements to generate one of these convolution feature ring elements, and the convolution feature components of each convolution feature ring element are obtained by performing component sum on multiple multiplication output components of these multiplication output ring elements. The deep neural network acceleration system using the ring tensor is applied to images.

[0013] Therefore, the deep neural network acceleration system using ring tensors of the present invention utilizes a directional nonlinear activation function, and can place additional linear transformations in the directional nonlinear activation function, thereby not increasing the number of bits of the multiplier in the ring element multiplication operation step. Furthermore, the present invention can effectively exchange information between channels and achieve the highest image quality in the ring algebra.

[0014] Other embodiments of the aforementioned implementation are as follows: The aforementioned directional nonlinear activation function may include a first conversion step, a linear rectification function operation step, and a second conversion step. The first conversion step is to perform a first linear conversion on one of these convolution feature ring elements to generate a linear output ring element. The linear rectification function operation step is to perform a component linear rectification function on the linear output ring element to generate a linear rectification output ring element. The second conversion step is to perform a second linear conversion on the linear rectification output ring element to generate an output feature ring element.

[0015] Other examples of the aforementioned implementation are as follows: either the aforementioned first linear transformation or the second linear transformation may be a Hadamard transformation.

[0016] Other embodiments of the aforementioned implementation are as follows: The aforementioned directional nonlinear activation function may include a first conversion step, a normalized linear conversion step, a linear rectification function operation step, and a second conversion step. The first conversion step is to perform a first linear conversion on one of these convolution feature ring elements to generate a first linear output ring element. The normalized linear conversion step is to perform ring element multiplication on the first linear output ring element and the normalized linear conversion to generate a second linear output ring element. The linear rectification function operation step is to perform a component linear rectification function on the second linear output ring element to generate a linear rectification output ring element. The second conversion step is to perform a second linear conversion on the linear rectification output ring element to generate an output feature ring element. Either the first linear conversion or the second linear conversion is a Hadamard conversion.

[0017] Other embodiments of the aforementioned implementation are as follows: the aforementioned operation processing unit is configured to implement a tensor quantization step, which is to perform a quantization operation on the output feature ring element to generate a quantized output ring element. The ring tensor setting step, the ring tensor convolution operation step, the tensor nonlinear activation function operation step and the tensor quantization step are performed in sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic flow chart showing a deep neural network acceleration method using a ring tensor according to a first embodiment of the present invention;

[0019] Figure 2 It is shown Figure 1 Schematic diagram of the ring element multiplication operation steps of the deep neural network acceleration method using ring tensor;

[0020] Figure 3 It is shown Figure 1 Schematic diagram of the ring element addition operation steps of the deep neural network acceleration method using ring tensor;

[0021] Figure 4 It is shown Figure 1 Flowchart of the directional nonlinear activation function of the deep neural network acceleration method using ring tensors;

[0022] Figure 5 is a schematic flow chart showing a deep neural network acceleration method using a ring tensor according to a second embodiment of the present invention;

[0023] Figure 6 is a schematic flow chart showing a method for accelerating a deep neural network using a ring tensor according to a third embodiment of the present invention;

[0024] Figure 7 is a schematic diagram showing a convolutional layer of a deep neural network acceleration method using a ring tensor according to a fourth embodiment of the present invention;

[0025] Figure 8 is a block diagram showing a deep neural network acceleration system using a ring tensor according to a fifth embodiment of the present invention; and

[0026] Fig. 9 It is a schematic diagram showing the comparison result between the present invention and the prior art.

[0027] Description of reference numerals:

[0028] 100, 100a, 100b, 100c: Deep neural network acceleration method using ring tensors

[0029] S02, S12: Ring tensor setting steps

[0030] S04, S14, S24: Ring tensor convolution operation steps

[0031] S042, S142: Ring element multiplication steps

[0032] S0422: Component Product

[0033] S044, S144: Ring element addition operation steps

[0034] S0442: Quantity and

[0035] S06, S16, S26: Tensor nonlinear activation function operation steps

[0036] S062, S162: Directional nonlinear activation function

[0037] S0622, S1622: First conversion step

[0038] S0624, S1626: Linear rectification function operation steps

[0039] S0626, S1628: Second conversion step

[0040] S08, S28: Tensor quantization steps

[0041] S082: Ring element quantization step

[0042] S1624: Regularized linear transformation step

[0043] S25: Deviation tensor superposition step 110: Input characteristic ring tensor

[0044] 120: Parameter Ring Tensor

[0045] 130: Convolutional feature ring tensor

[0046] x: Input characteristic ring element

[0047] x (l-1) 、x (l) : Input characteristic ring tensor

[0048] g: parameter ring element

[0049] g (l) : parameter ring tensor

[0050] cz: convolution feature ring element

[0051] cz (l) : Convolution feature ring tensor

[0052] z: multiplication output ring element

[0053] T x : Input feature transformation matrix

[0054] T g : parameter transformation matrix

[0055] T z : Multiplication output conversion matrix

[0056] Input feature conversion output

[0057] Parameter conversion output

[0058] Multiplication Output

[0059] b (l) : Deviation ring tensor

[0060] 200: Deep Neural Network Acceleration System Using Ring Tensors

[0061] 210: First Memory

[0062] 220: Second memory

[0063] 230: Processing unit

[0064] Real, R H2 , R C , R H4 , R F4 , R Q , (R I2 、f H2 )、(R I4 、f H4 ):symbol

[0065] PSNR: Peak Signal-to-Noise Ratio DETAILED DESCRIPTION

[0066] The following will describe multiple embodiments of the present invention with reference to the accompanying drawings. For the purpose of clarity, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is, in some embodiments of the present invention, these practical details are not necessary. In addition, in order to simplify the drawings, some conventional structures and elements will be shown in the drawings in a simple schematic manner; and repeated elements may be represented by the same number.

[0067] In addition, in this article, when a certain element (or unit or module, etc.) is "connected / linked" to another element, it may refer to that the element is directly connected / linked to another element, or it may refer to that a certain element is indirectly connected / linked to another element, that is, there are other elements between the element and the other element. When it is clearly stated that a certain element is "directly connected / linked" to another element, it means that there are no other elements between the element and the other element. The terms first, second, third, etc. are only used to describe different elements, and there is no restriction on the elements themselves. Therefore, the first element can also be renamed as the second element. Moreover, the combination of elements / units / circuits in this article is not a generally known, conventional or existing combination in this field. Whether the elements / units / circuits themselves are existing cannot be used to determine whether their combination relationship is easy to be easily completed by those with ordinary knowledge in the technical field.

[0068] See also Figure 1 , Figure 1 1 is a flow chart showing a deep neural network acceleration method 100 using a ring tensor according to a first embodiment of the present invention. The deep neural network acceleration method 100 using a ring tensor comprises a ring tensor setting step S02, a ring tensor convolution operation step S04, and a tensor nonlinear activation function operation step S06.

[0069] The ring tensor setting step S02 sets the input feature ring tensor 110 (input feature tensor) and the parameter ring tensor 120 (weight tensor) of the convolution network. The input feature ring tensor 110 includes a plurality of input feature ring elements, and the parameter ring tensor 120 includes a plurality of parameter ring elements. The ring tensor convolution operation step S04 operates the input feature ring elements of the input feature ring tensor 110 and the parameter ring elements of the parameter ring tensor 120 according to the ring element multiplication operation step S042 and the ring element addition operation step S044 to generate a plurality of convolution feature ring elements of the convolution feature ring tensor 130. The ring element multiplication operation step S042 includes performing ring element multiplication on one of the input feature ring elements and one of the parameter ring elements to generate a multiplication output ring element. A multiplication output component of the multiplication output ring element is obtained by performing component-wise product on the input feature component of one of the input feature ring elements and the parameter component of one of the parameter ring elements. The ring element addition operation step S044 includes performing a ring element addition on a plurality of multiplication output ring elements to generate one of these convolution feature ring elements, and a convolution feature component of each convolution feature ring element is obtained by performing component-wise addition on a plurality of multiplication output components of these multiplication output ring elements. In addition, the tensor nonlinear activation function operation step S06 performs a directional nonlinear activation function S062 (directional Rectified Linear Unit; directional ReLU) on one of these convolution feature ring elements of the convolution feature ring tensor 130 to generate an output feature ring element. Therefore, the deep neural network acceleration method 100 using a ring tensor of the present invention utilizes a directional nonlinear activation function to maintain image quality without increasing the number of bits of the multiplier in the ring element multiplication operation step S042, thereby avoiding the problem of increased complexity caused by increasing the number of bits of the multiplier in the prior art. The details of the above steps will be described below through a more detailed embodiment.

[0070] Please also read Figures 1 to 4 ,in Figure 2 It is shown Figure 1 A schematic diagram of the ring element multiplication operation step S042 of the deep neural network acceleration method 100 using the ring tensor; Figure 3 It is shown Figure 1 A schematic diagram of the ring element addition operation step S044 of the deep neural network acceleration method 100 using a ring tensor; and Figure 4 It is shown Figure 1The flowchart of the directional nonlinear activation function S062 of the deep neural network acceleration method 100 using the ring tensor is shown in FIG. As shown in the figure, the deep neural network acceleration method 100 using the ring tensor includes a ring tensor setting step S02, a ring tensor convolution operation step S04, and a tensor nonlinear activation function operation step S06.

[0071] The ring tensor setting step S02 is to set the input feature ring tensor 110 and parameter ring tensor 120 of the convolutional network. The input feature ring tensor 110 includes multiple input feature ring elements x, and the parameter ring tensor 120 includes multiple parameter ring elements g. The ring tensor is a multi-dimensional array, which can be a ring vector or a ring matrix. The ring element is an n-dimensional real vector, where n is a positive integer greater than or equal to 2. The input feature ring element x includes multiple input feature components x i , i is an integer greater than or equal to 0. The parameter ring element g contains multiple parameter components g i . Input feature component x i and parameter component g i All are real numbers.

[0072] The ring tensor convolution operation step S04 operates the input feature ring element x of the input feature ring tensor 110 and the parameter ring element g of the parameter ring tensor 120 according to the ring element multiplication operation step S042 and the ring element addition operation step S044 to generate a plurality of convolution feature ring elements cz of the convolution feature ring tensor 130. The convolution feature ring element cz includes a plurality of convolution feature components cz j .

[0073] The ring element multiplication operation step S042 includes performing a ring element multiplication on one of the input feature ring elements x and one of the parameter ring elements g to generate a multiplication output ring element z. A multiplication output component z of the multiplication output ring element z i An input feature component x of one of the input feature ring elements x i and a parameter component g of one of these parameter ring elements g i Execute a component product S0422 to obtain, such as Figure 2 Specifically, the ring element multiplication step S042 includes firstly passing one of the input feature ring elements x and one of the parameter ring elements g through the input feature transformation matrix T x With the parameter transformation matrix T g Transformation produces input feature transformation output Convert output with parameters Then transform the input features into output Convert output with parameters Perform component product S0422 to produce multiplication output Finally, the multiplication output Output the transformation matrix T by multiplication z The input feature conversion matrix T of this embodiment is converted to generate the multiplication output ring element z. x , parameter transformation matrix T g And the multiplication output transformation matrix T z The relationship between the multiplication output ring element z and the input characteristic ring element x can be expressed by the following equation (1):

[0074] z=Gx (1).

[0075] Where G represents the isomorphic matrix, and this isomorphic matrix G and parameter component g i There is a connection.

[0076] The ring element addition operation step S044 includes performing a ring element addition (ring addition) on the plurality of multiplication output ring elements z to generate one of the convolution feature ring elements cz, a convolution feature component cz of each convolution feature ring element cz j The multiple multiplication output components z of these multiplication output ring elements z i Execute a component and S0442 to obtain, such as Figure 3 shown.

[0077] The tensor nonlinear activation function operation step S06 is to perform a directional nonlinear activation function S062 on these convolution feature ring elements cz of the convolution feature ring tensor 130 to generate output feature ring elements f dir (cz). In detail, the directional nonlinear activation function S062 includes a first conversion step S0622, a linear rectification function operation step S0624 and a second conversion step S0626. The first conversion step S0622 performs a first linear transformation V on these convolution feature ring elements cz to generate a linear output ring element Vcz. The linear rectification function operation step S0624 performs a component linear rectification function f on the linear output ring element Vcz. cw (component-wise ReLU) produces a linear rectified output loop element f cw (Vcz). The second conversion step S0626 is to convert the linear rectifier output loop element f cw (Vcz) performs a second linear transformation U to generate an output characteristic ring element f dir(cz). Furthermore, the ring tensor setting step S02, the ring tensor convolution operation step S04 and the tensor nonlinear activation function operation step S06 are performed in sequence, and the first conversion step S0622, the linear rectification function operation step S0624 and the second conversion step S0626 are performed in sequence. The component linear rectification function f cw and the output characteristic ring element f dir (cz) respectively conform to the following equations (2) and (3):

[0078] f cw (y)=(max(0,y0),…,max(0,y n-1 )) t (2);

[0079] f dir (cz)=Uf cw (Vcz) (3).

[0080] Where y represents the ring element, y0~y n-1 Represents the component y of the ring element y i i is 0 to n-1. In this embodiment, either the first linear transformation V or the second linear transformation U can be a Hadamard Transform H, and the corresponding output characteristic ring element can be represented by "f H (cz)” and meets the following formula (4):

[0081] f H (cz)=Hf cw (Hcz) (4).

[0082] Therefore, the deep neural network acceleration method 100 using ring tensors of the present invention can place the additional first linear transformation V and the second linear transformation U in the directional nonlinear activation function S062 through the directional nonlinear activation function, so as not to increase the number of bits of the multiplier in the ring element multiplication operation step S042. In addition, the present invention can effectively exchange information between channels and achieve the highest image quality in the ring algebra.

[0083] Please also read Figures 1 to 5 ,in Figure 51 is a flow chart showing a deep neural network acceleration method 100a using a ring tensor according to a second embodiment of the present invention. As shown in the figure, the deep neural network acceleration method 100a using a ring tensor includes a ring tensor setting step S12, a ring tensor convolution operation step S14, and a tensor nonlinear activation function operation step S16. The ring tensor setting step S12 includes setting the input feature ring tensor 110 and the parameter ring tensor 120 of the convolutional network. The ring tensor convolution operation step S14 includes a ring element multiplication operation step S142 and a ring element addition operation step S144. The ring tensor setting step S12 and the ring tensor convolution operation step S14 are respectively Figure 1 The operations of the ring tensor setting step S02 and the ring tensor convolution operation step S04 are the same, and the details are not repeated. In particular, Figure 5 The tensor nonlinear activation function operation step S16 is to perform a directional nonlinear activation function S162 on one of the convolution feature ring elements cz of the convolution feature ring tensor 130 to generate an output feature ring element f dir (cz). The directional nonlinear activation function S162 includes a first conversion step S1622, a normalized linear conversion step S1624, a linear rectification function operation step S1626, and a second conversion step S1628. The first conversion step S1622 performs a first linear conversion V on one of these convolution feature ring elements cz to generate a first linear output ring element Vcz. The normalized linear conversion step S1624 performs a ring element multiplication (same as Figure 2 The linear rectification function operation step S1626 performs a component linear rectification function f on the second linear output ring element VczT. cw The linear rectifier output loop element f cw (VczT). The second conversion step S1628 is to convert the linear rectifier output loop element f cw (VczT) performs the second linear transformation U to produce the output characteristic ring element f dir (cz). Either the first linear transformation V or the second linear transformation U of the present embodiment may be a Hadamard transformation H, and the normalized linear transformation T may be a normalized diagonal matrix for normalizing the components of the ring elements, but the present invention is not limited thereto. Therefore, the deep neural network acceleration method 100a using the ring tensor of the present invention may utilize the normalized linear transformation T to achieve the required transformation, which, in combination with a directional nonlinear activation function, can place the additional first linear transformation V and the second linear transformation U in the directional nonlinear activation function S162, thereby not increasing the number of bits of the multiplier in the ring element multiplication operation step S142. In addition, the present invention can effectively exchange information between channels and achieve the highest image quality in the ring algebra.

[0084] Please also read Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 6 ,in Figure 6 1 is a flow chart showing a method 100b for accelerating a deep neural network using a ring tensor according to a third embodiment of the present invention. As shown in the figure, the method 100b for accelerating a deep neural network using a ring tensor comprises a ring tensor setting step S02, a ring tensor convolution operation step S04, a tensor nonlinear activation function operation step S06, and a tensor quantization step S08, wherein the ring tensor setting step S02, the ring tensor convolution operation step S04, and the tensor nonlinear activation function operation step S06 are respectively Figure 1 The operations of the ring tensor setting step S02, the ring tensor convolution operation step S04 and the tensor nonlinear activation function operation step S06 are the same, and the details are not repeated. In particular, Figure 6 The deep neural network acceleration method 100b using the ring tensor further comprises a tensor quantization step S08, wherein the tensor quantization step S08 comprises a ring element quantization step S082, wherein the ring element quantization step S082 is to quantize the output feature ring element f dir (cz) performs quantization operation Q to generate quantized output ring element Q(cz). Ring tensor setting step S02, ring tensor convolution operation step S04, tensor nonlinear activation function operation step S06 and tensor quantization step S08 are performed in sequence. Therefore, the deep neural network acceleration method 100b using ring tensor of the present invention can convert the output feature ring element f generated by the tensor nonlinear activation function operation step S06 into dir (cz) is quantized to obtain output features with a target number of bits.

[0085] Please also read Figure 1 , Figure 6 and Figure 7 ,in Figure 7 1 is a schematic diagram showing the convolution layer of the deep neural network acceleration method 100c using the ring tensor according to the fourth embodiment of the present invention. As shown in the figure, the convolution layer of the deep neural network acceleration method 100c using the ring tensor includes a ring tensor convolution operation step S24, a bias tensor superposition step S25, a tensor nonlinear activation function operation step S26, and a tensor quantization step S28. The ring tensor convolution operation step S24 is based on the ring element multiplication operation step (same as Figure 1 The ring element multiplication step S042) and the ring element addition step (same Figure 1 The ring element addition step S044) is performed on the input characteristic ring tensor x (l-1) and parameter ring tensor g (l) Perform operations to produce multiple convolution feature ring tensors cz(l) Where l represents the lth layer. The deviation tensor superposition step S25 is to convolution feature ring tensor cz (l) Superposition of a deviation ring tensor b (l) (bias) to produce the updated convolution feature ring tensor cz (l) , and then the updated convolution feature ring tensor cz (l) Input to the tensor nonlinear activation function operation step S26. After the tensor nonlinear activation function operation step S26 and the tensor quantization step S28, the quantized output loop tensor Q(cz) generated by the tensor quantization step S28 will be used as the input feature loop tensor x of the next layer. (l) It is also worth mentioning that, in other embodiments, the updated convolution feature ring tensor cz that will be input into the tensor nonlinear activation function operation step S26 (l) The number of can be determined according to the model structure; in other words, not necessarily every updated convolution feature ring tensor cz (l) Therefore, the convolution layer of the deep neural network acceleration method 100c using the ring tensor of the present invention can be obtained by using the deviation ring tensor b (l) Adjust the convolution feature ring tensor cz of the ring tensor convolution operation step S24 (l) , and then obtain the desired output features.

[0086] Please also read Figures 1 to 8 ,in Figure 8 2 is a block diagram showing a deep neural network acceleration system 200 using a ring tensor according to a fifth embodiment of the present invention. As shown in the figure, the deep neural network acceleration system 200 using a ring tensor includes a first memory 210, a second memory 220 and an operation processing unit 230.

[0087] The first memory 210 is used to access the input feature ring tensor 110 of the convolutional network, and the input feature ring tensor 110 includes a plurality of input feature ring elements x. The second memory 220 is used to access the parameter ring tensor 120 of the convolutional network, and the parameter ring tensor 120 includes a plurality of parameter ring elements g.

[0088] The operation processing unit 230 is electrically connected to the first memory 210 and the second memory 220. The operation processing unit 230 receives the input feature loop tensor 110 and the parameter loop tensor 120 and is configured to implement the loop tensor setting step S02, the loop tensor convolution operation step S04 and the tensor nonlinear activation function operation step S06. Figure 1 The processing unit 230 may be a microprocessor, a central processing unit, or an image processor, but the present invention is not limited thereto. In other embodiments, the processing unit 230 may be configured to implement Figure 5A method for accelerating deep neural networks using ring tensors 100a, Figure 6 A method for accelerating deep neural networks using ring tensors 100b and Figure 7 A method for accelerating deep neural networks using ring tensors 100c.

[0089] Therefore, the deep neural network acceleration system 200 using ring tensors of the present invention utilizes a directional nonlinear activation function, and can place additional linear transformations in the directional nonlinear activation functions S062 and S162, thereby not increasing the number of bits of the multiplier in the ring element multiplication operation step S042. Furthermore, the present invention can effectively exchange information between channels and achieve the highest image quality in the ring algebra.

[0090] Please refer to Table 1, Table 2, Table 3 and Fig. 9 , wherein Table 1 shows the algebraic properties of a ring with dimension D (Dimension) 2; Table 2 shows a part of the algebraic properties of a ring with dimension D 4; Table 3 shows another part of the algebraic properties of a ring with dimension D 4; and Fig. 9 : is a schematic diagram showing the comparison results of the present invention and the prior art. The symbol Real represents the convolution operation and activation function operation of the original real number. The symbol R H2 , R H4 Represents the input feature transformation matrix T x With the parameter transformation matrix T g are Hadamard transformations H2 and H4, which conform to the following equations (5) and (6) respectively:

[0091]

[0092] Symbol R C The isomorphism matrix G is a 2×2 rotation matrix, and the input feature transformation matrix T x With the parameter transformation matrix T g It is derived from a complex multiplication algorithm; symbol R I2 , R I4 Represents the isomorphism matrix G using component products, and the input feature transformation matrix T x With the parameter transformation matrix T g is the unit matrix; symbol R F4 Represents the isomorphic matrix G using circular convolution, and the input feature transformation matrix T x With the parameter transformation matrix T g It is derived from Fourier transform. Symbol R QThe quaternion representing the homogeneous matrix G is Hamiltonian. Symbol R H2 , R C , R I2 , R H4 , R F4 , R Q , R I4 It is the existing technology, and its nonlinear function f is the component linear rectification function f cw ; Symbol (R I2 ,f H2 )、(R I4 ,f H4 ) is the present invention, and its nonlinear function f is a directional nonlinear activation function f H2 (x), f H4 (x). Fig. 9 The comparison results are obtained by simulating the four-times super-resolution network model SR4ERNet-B17R3N1 (four-times super-resolution (SR×4) networks SR4ERNet). From Table 1, Table 2, Table 3 and Fig. 9 It can be seen that (R I2 ,f H2 )、(R I4 ,f H4 ) uses a directional nonlinear activation function. Compared with the existing ring algebra, the present invention requires the least hardware (i.e., the lowest hardware resource ratio and area ratio) and provides the highest quality (i.e., the highest Peak Signal-to-Noise Ratio (PSNR)) without increasing the number of bits of the multiplier.

[0093] Table 1

[0094]

[0095]

[0096] Table 2

[0097]

[0098]

[0099] Table 3

[0100]

[0101] It can be seen from the above embodiments that the present invention has the following advantages: First, by using a directional nonlinear activation function, the image quality is maintained without increasing the number of bits of the multiplier in the ring element multiplication operation step, which can avoid the problem of increased complexity caused by increasing the number of bits of the multiplier in the prior art. Second, by using a directional nonlinear activation function, additional linear transformations can be placed in the directional nonlinear activation function, so the number of bits of the multiplier in the ring element multiplication operation step will not be increased. Third, the present invention can effectively exchange information between channels and achieve the highest image quality in the ring algebra.

[0102] Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A method for accelerating deep neural networks using ring tensors, Features The following steps are involved: A ring tensor setting step is to set an input feature ring tensor and a parameter ring tensor of a convolutional network, wherein the input feature ring tensor includes a plurality of input feature ring elements, and the parameter ring tensor includes a plurality of parameter ring elements; A ring tensor convolution operation step is to operate the input feature ring elements of the input feature ring tensor and the parameter ring elements of the parameter ring tensor according to a ring element multiplication operation step and a ring element addition operation step to generate a plurality of convolution feature ring elements of a convolution feature ring tensor; as well as A tensor nonlinear activation function operation step is to perform a directional nonlinear activation function on one of the convolution feature ring elements of the convolution feature ring tensor to generate an output feature ring element; The ring element multiplication operation step includes performing a ring element multiplication on one of the input characteristic ring elements and one of the parameter ring elements to generate a multiplication output ring element, wherein a multiplication output component of the multiplication output ring element is obtained by performing a component product on an input characteristic component of the one of the input characteristic ring elements and a parameter component of the one of the parameter ring elements; The ring element addition operation step comprises performing a ring element addition on a plurality of the multiplication output ring elements to generate the convolution feature ring elements, and a convolution feature component of each of the convolution feature ring elements is obtained by performing a component sum on a plurality of the multiplication output components of the multiplication output ring elements; Among them, the deep neural network acceleration method using ring tensors is applied to images.

2. The deep neural network acceleration method using ring tensor as claimed in claim 1, Features The directional nonlinear activation function contains: A first conversion step is to perform a first linear conversion on the one of the convolution feature ring elements to generate a linear output ring element; A linear rectification function operation step is to perform a component linear rectification function on the linear output loop element to generate a linear rectification output loop element; and A second conversion step is to perform a second linear conversion on the linear rectified output loop element to generate the output characteristic loop element.

3. The deep neural network acceleration method using ring tensor as claimed in claim 2, Features Either the first linear transformation or the second linear transformation is a Hadamard transformation.

4. The deep neural network acceleration method using ring tensor as claimed in claim 1, Features The directional nonlinear activation function contains: A first conversion step is to perform a first linear conversion on the one of the convolution feature ring elements to generate a first linear output ring element; A normalized linear transformation step is to perform a ring element multiplication on the first linear output ring element and a normalized linear transformation to generate a second linear output ring element; a linear rectification function operation step, which performs a component linear rectification function on the second linear output loop element to generate a linear rectification output loop element; and a second conversion step, performing a second linear conversion on the linear rectifier output loop element to generate the output characteristic loop element; Wherein, either the first linear transformation or the second linear transformation is a Hadamard transformation.

5. The deep neural network acceleration method using ring tensor as claimed in claim 1, Features Also includes: A quantization step is to perform a quantization operation on the output characteristic ring element to generate a quantized output ring element; Among them, the ring tensor setting step, the ring tensor convolution operation step, the tensor nonlinear activation function operation step and the tensor quantization step are performed in sequence.

6. A deep neural network acceleration system using ring tensors, Features Include: A first memory for accessing an input feature ring tensor of a convolutional network, wherein the input feature ring tensor comprises a plurality of input feature ring elements; a second memory for accessing a parameter ring tensor of the convolutional network, wherein the parameter ring tensor comprises a plurality of parameter ring elements; as well as A processing unit is electrically connected to the first memory and the second memory, and receives the input feature loop tensor and the parameter loop tensor and is configured to perform operations including the following steps: A ring tensor setting step is to set the input feature ring tensor and the parameter ring tensor of the convolutional network; A ring tensor convolution operation step is to operate the input feature ring elements of the input feature ring tensor and the parameter ring elements of the parameter ring tensor according to a ring element multiplication operation step and a ring element addition operation step to generate a plurality of convolution feature ring elements of a convolution feature ring tensor; and A tensor nonlinear activation function operation step is to perform a directional nonlinear activation function on one of the convolution feature ring elements of the convolution feature ring tensor to generate an output feature ring element; The ring element multiplication operation step includes performing a ring element multiplication on one of the input characteristic ring elements and one of the parameter ring elements to generate a multiplication output ring element, wherein a multiplication output component of the multiplication output ring element is obtained by performing a component product on an input characteristic component of the one of the input characteristic ring elements and a parameter component of the one of the parameter ring elements; The ring element addition operation step comprises performing a ring element addition on a plurality of the multiplication output ring elements to generate the convolution feature ring elements, and a convolution feature component of each of the convolution feature ring elements is obtained by performing a component sum on a plurality of the multiplication output components of the multiplication output ring elements; Among them, the deep neural network acceleration system using ring tensors is applied to images.

7. The deep neural network acceleration system using ring tensor as claimed in claim 6, Features The directional nonlinear activation function contains: A first conversion step is to perform a first linear conversion on the one of the convolution feature ring elements to generate a linear output ring element; A linear rectification function operation step is to perform a component linear rectification function on the linear output loop element to generate a linear rectification output loop element; and A second conversion step is to perform a second linear conversion on the linear rectified output loop element to generate the output characteristic loop element.

8. The deep neural network acceleration system using ring tensor as claimed in claim 7, Features Either the first linear transformation or the second linear transformation is a Hadamard transformation.

9. The deep neural network acceleration system using ring tensor as claimed in claim 6, Features The directional nonlinear activation function contains: A first conversion step is to perform a first linear conversion on the one of the convolution feature ring elements to generate a first linear output ring element; A normalized linear transformation step is to perform a ring element multiplication on the first linear output ring element and a normalized linear transformation to generate a second linear output ring element; a linear rectification function operation step, which performs a component linear rectification function on the second linear output loop element to generate a linear rectification output loop element; and a second conversion step, performing a second linear conversion on the linear rectifier output loop element to generate the output characteristic loop element; Wherein, either the first linear transformation or the second linear transformation is a Hadamard transformation.

10. The deep neural network acceleration system using ring tensor as claimed in claim 6, Features The operation processing unit is configured to implement a tensor quantization step, wherein the tensor quantization step is to perform a quantization operation on the output feature ring element to generate a quantized output ring element; Among them, the ring tensor setting step, the ring tensor convolution operation step, the tensor nonlinear activation function operation step and the tensor quantization step are performed in sequence.

Citation Information

Patent Citations

  • A method for efficiently tensioning a fully connected neural network

    CN109344966A

  • Subset conditioning using variational autoencoder with a learnable tensor train induced prior

    US20200090049A1

  • Neural network processor using dyadic weight matrix and operation method thereof

    US20200167637A1

  • Accelerator and system for accelerating operations

    WO2019201657A1