Method for optimizing neural network model in optical computing

Abstract

The present invention provides a method for optimizing a neural network model in optical computing. The method comprises: acquiring an initial quantization scaling coefficient and an optical computing system noise suppression coefficient; adjusting the initial quantization coefficient on the basis of the noise suppression coefficient; and then using the adjusted coefficient to quantize a neural network model. The present invention improves the stability and reliability of a model in an optical computing system, thereby reducing the impact of noise of the optical computing system on a neural network model.

Description

Optimization method of neural network model in optical computing

[0001] The present application claims priority to the Chinese patent application No. 202411402311.0, filed on September 30, 2024, and entitled "Optimization method of neural network model in optical computing", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of data processing, and particularly relates to an optimization method of neural network model in optical computing. BACKGROUND

[0003] In optical computing, due to the reason of optical computing system, there may be noise which will affect the calculation result. SUMMARY

[0004] The present application provides an optimization method of neural network model in optical computing, aiming at solving the problem of how to reduce the influence of optical computing system noise on neural network model.

[0005] In a first aspect, the present application provides an optimization method of neural network model in optical computing, the method comprising:

[0006] obtaining an initial quantization scaling factor;

[0007] obtaining a noise suppression coefficient related to the optical computing system;

[0008] adjusting the initial quantization scaling factor according to the noise suppression coefficient to obtain an adjusted quantization scaling factor;

[0009] quantizing the neural network model using the adjusted quantization scaling factor.

[0010] The present application provides an optimization method of neural network model in optical computing, aiming at reducing the influence of optical computing system noise on neural network model. The method first obtains an initial quantization scaling factor, then obtains a noise suppression coefficient related to the optical computing system, which reflects the characteristics of noise in the optical computing system. Then the initial quantization scaling factor is adjusted according to the noise suppression coefficient to obtain an adjusted quantization scaling factor. Since the adjusted quantization scaling factor takes into account the noise of the optical computing system, when it is used to quantize the neural network model, the model can adapt to the noise environment of the optical computing system in the quantization process. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0012] Fig. 1 is a flowchart of the method for optimizing the neural network model in optical computing provided by the present application;

[0013] Fig. 2 is a flowchart of quantization-dequantization without considering noise;

[0014] Fig. 3 is a flowchart of the calculation of the noise suppression coefficient of activation provided by the embodiment of the present application;

[0015] Fig. 4 is a flowchart of the calculation of the noise suppression coefficient of weight provided by the embodiment of the present application;

[0016] Fig. 5 is a schematic diagram of optical computing provided by the embodiment of the present application;

[0017] Fig. 6 is a flowchart of the method for optimizing the neural network model in optical computing provided by the embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application.

[0019] The terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0020] In this paper, the mentioned noun vector, matrix, belongs to tensor. Tensor can have one-dimensional form (vector), two-dimensional form (matrix), and also can be 3-dimensional and above-dimensional tensor. Generally, the tensor of 3-dimensional and above is no longer named as matrix, but in this paper, for the convenience of representation, when referring to the shape of matrix, in combination with the context, it represents the shape of 3-dimensional tensor without causing confusion; in addition, vector can exist as a special case of matrix.

[0021] The optimization method of the neural network model in optical computing of the present invention is described below with reference to Figures 1-6.

[0022] Please refer to Figure 1, which is a schematic diagram of the optimization method for a neural network model in optical computing provided by this invention. The optimization method for a neural network model in optical computing includes:

[0023] S110, obtain the initial quantization scaling factor.

[0024] S120, obtains the noise suppression coefficient related to the optical computing system.

[0025] S130, adjust the initial quantization scaling factor according to the noise suppression factor to obtain the adjusted quantization scaling factor.

[0026] S140 uses the adjusted quantization scaling factor to quantize the neural network model.

[0027] The following is a detailed description of steps S110 to S140.

[0028] In some embodiments, step S110, obtaining the initial quantization scaling factor includes:

[0029] S111, determine the initial quantization scaling factor based on the maximum and minimum values ​​of the floating-point numbers corresponding to the data to be quantized.

[0030] Specifically, quantization includes symmetric quantization and asymmetric quantization. For example, the quantization described above can be performed in the following manner, and correspondingly, the initial quantization scaling factor can be obtained from the following "quantization scaling factor":

[0031] (1) Symmetric quantization:

[0032] Symmetric quantization is the process of converting floating-point numbers to values ​​within the range of [-2]. n-1 -2 n-1 A signed integer (integral numeric type, INT) of [-1], where n is the number of bits for symmetric quantization. Assume the floating-point number to be quantized is x. f The quantitative objective is to make x f Perform n-bit symmetric quantization, meaning the range of the quantized fixed-point number is x. q The symmetric quantization process is expressed by the following formula (1).

[0033] In formula (1), abs() is the absolute value function, max() is the maximum value function, min() is the minimum value function, round is the rounding function, and x qThe fixed-point number. In addition, in the formula (1), the Sc term can be referred to as a symmetric quantization scaling coefficient as shown in the following formula (2). That is, for a fixed-point number corresponding to a symmetric quantization, the floating-point number corresponding to the fixed-point number can be determined according to the symmetric quantization scaling coefficient corresponding to the fixed-point number.

[0034] (2) Asymmetric quantization

[0035] Asymmetric quantization is to convert a floating-point number into an unsigned integral numeric type (UINT) number with a value range of [0, 2 n -1], where n is the number of bits of asymmetric quantization. Assuming that the floating-point number to be quantized is x f , and the quantization target is to perform n-bit asymmetric quantization on x f , that is, the value range of the fixed-point number after quantization is [0, 2 n -1], the asymmetric quantization process is represented as the following formula (3).

[0036] In the formula (3), max() is a maximum value function, min() is a minimum value function, round is a rounding function, and x q is a fixed-point number. In addition, in the formula (3), the AS c term can be referred to as an asymmetric quantization scaling coefficient as shown in the following formula (4), and the Z p term can be referred to as an asymmetric quantization zero point as shown in the following formula (5). That is, for a fixed-point number corresponding to asymmetric quantization, the floating-point number corresponding to the fixed-point number can be determined according to the asymmetric quantization scaling coefficient and the asymmetric quantization zero point corresponding to the fixed-point number. Z P = round(-AS c x f ) (5)

[0037] Based on the formula (3) to the formula (5), another representation of asymmetric quantization can be obtained as shown in the formula (6): x q = round(x f AS c + Z P ) (6)

[0038] In order to compare the model calculation with noise suppression and the model calculation without noise, the model without noise suppression coefficient can be understood based on Figure 2 first, and then the neural network model in the light calculation with noise suppression coefficient will be further described according to Figure 6.

[0039] In the example, the shape of the activation matrix X is (1, 2, 4), which means that the activation matrix qX to be quantized has 1 batch of data, in each batch, there are 2 groups of features, and each group has 4 elements, i.e., 2 rows and 4 columns.

[0040] For example, when the shape of the activation matrix X is (1, 2, 4), it means that the activation matrix qX to be quantized has 1 batch of data, in each batch, there are 2 groups of features, and each group has 4 elements, i.e., 2 rows and 4 columns. For example, the activation matrix X is:

[0041] [[[1.33, 0.72, -1.55, -0.01], [0.62, -0.72, 0.27, 0.11]]], i.e.,

[0042] The activation matrix X indicates that the corresponding batch dimension B = 1, and the two sub-lists [1.33, 0.72, -1.55, -0.01] and [0.62, -0.72, 0.27, 0.11] are two groups. Each group has 4 elements, for example, for the group [1.33, 0.72, -1.55, -0.01], 1.33, 0.72, -1.55, and -0.01 are the 4 elements in this group.

[0043] In this example, if the noise suppression coefficient is not considered, the quantized activation matrix qX is:

[0044] [[[3, 1, -3, 0], [3, -3, 1, 0]]], i.e.,

[0045] The shape of the quantized activation matrix qX is still (1, 2, 4), which means that the quantization operation does not change the shape structure of the matrix.

[0046] Exemplarily, the activation quantization scaling coefficient xS is represented according to the third dimension (i.e., the K dimension), and the matrix (tensor) of the activation quantization scaling system xS corresponds to the shape (1, 2, 1).

[0047] For example, quantization is performed according to the dimension K = 1, and the following is obtained:

[0048] xS = [[[0.52], [0.24]]], i.e.,

[0049] In which, 0.52 corresponds to [1.33, 0.72, -1.55, -0.01], and 0.24 corresponds to [0.62, -0.72, 0.27, 0.11].

[0050] The method further includes performing a shape conversion operation on the activation matrix X to be quantized, and the like, specifically including:

[0051] (1) performing a shape conversion operation on the activation matrix X to be quantized to convert its shape from (B, M, K) to (B x M, , Ks), where denotes rounding up; if K cannot be divided by Ks, the remainder is padded with 0. Where Ks represents the number of summation objects in a point multiplication summation unit in optical computing hardware (such as an optical computing unit), if the summation unit supports summation sum = a1.b1 + a2.b2 + a3.b3, then Ks = 3, where a1.b1 represents the product of a1 and b1 (i.e., a summation object), and the other a2.b2, a3.b3 have the same meaning, and Ks = 3 indicates that the summation unit in the hardware is designed to include summation of 3 summation objects, if the summation unit is sum = a1.b1 + a2.b2 + a3.b3 + a4.b4, then Ks = 4; Obviously, if the model calculation needs to sum more objects, the above shape conversion operation can be used to adapt to the hardware.

[0052] Specifically, in matrix operations, the above shape conversion (reshape, or called deformation) operation is an operation that changes the shape of the matrix, which does not change the element values in the matrix, but only rearranges the layout of the elements. Here, the shape of qX is converted from (B, M, K) to (B x M, , Ks).

[0053] For example, for a given X, the original shape is (1, 2, 4), when Ks = 2, the elements in qX are rearranged according to the reshape operation rule. That is, the original X = [[1.33, 0.72, -1.55, -0.01], [0.62, -0.72, 0.27, 0.11]]] becomes [[1.33, 0.72], [-1.55, -0.01]] after shape conversion, [0.62, -0.72], [0.27, 0.11]]], the shape becomes (2, 2, 2). Here, the original 1 x 2 x 4 matrix is rearranged into a 2 x 2 x 2 matrix.

[0054] (2) performing a matrix transposition operation on the activation matrix X to convert it from (B x M, , Ks) to (B x M, , Ks).

[0055] Specifically, the transposition (permute) operation is to rearrange the dimensions of the matrix, that is, the transposition operation. Here, the shape of the X after the shape conversion is converted from (B x M,

[0056] For example, after the shape conversion, the shape of the former is (2, 2, 2), and after the transposition operation, the shape is still (2, 2, 2), but the position of the elements in different dimensions has changed.

[0057] (3) According to the activation quantization scaling coefficient xS, the transposed activation matrix is quantized;

[0058] Wherein, the activation quantization scaling coefficient xS corresponding to each activation part can be extracted along the K dimension, and the shape of the activation quantization scaling coefficient xS is (Ks, B x M, 1).

[0059] Specifically, the activation quantization scaling coefficient xS of each part is obtained along the K dimension, and it is represented as a matrix with a shape of (Ks, B x M, 1).

[0060] For example, xS originally is [[[0.52], [0.24]]] with a shape of (1, 2, 1), and after the operation, it becomes xS = [[[0.52], [0.24]], [[0.52], [0.24]]] with a shape of (2, 2, 1).

[0061] For example, after quantizing the transposed X, qX = [[[3, 1], [3, -3]], [[-3, 0], [1, 0]]] can be obtained.

[0062] Exemplarily, the step S112 further comprises:

[0063] S1122, performing a shape conversion operation on the weight matrix W, wherein the shape of the weight matrix W is (K, N); and quantizing the weight matrix W according to the weight quantization scaling coefficient wS to obtain the quantized weight matrix qW.

[0064] For example, when the shape of the weight matrix W to be quantized is (4, 2), it means that the quantized weight matrix qW has 4 rows and 2 columns. Taking W = [[0, -0.17], [0.43, 1.20], [-0.97, 1.03], [0.23, 0.45]] as an example, that is,

[0065] The shape conversion operation is performed on the weight matrix W to convert its shape from (K, N) to (Ks, K, N), wherein Ks represents the number of summation objects in the point multiplication summation unit of the optical computing hardware, ​​​​​denotes the ceiling operation; if K is not divisible by Ks, the elements of the remainder are padded with 0 to represent the corresponding elements.

[0066] Specifically, a shape conversion operation is performed on the weight matrix W to convert its shape from (K, N) to (K, N, Ks). , K, N).

[0067] For example, for the above W = [[0, -0.17], [0.43, 1.20], [-0.97, 1.03], [0.23, 0.45]], the shape is (4, 2). After the shape conversion operation, the shape becomes (2, 2, 2) and W becomes [[[0, -0.17], [0.43, 1.20]], [[-0.97, 1.03], [0.23, 0.45]]].

[0068] After the shape conversion, the quantization results in [[[0, 0], [1, 3]], [[-3, 3], [1, 1]]].

[0069] In the quantization process, the weight quantization scaling factor wS can be represented in the form of a corresponding matrix, and the weight quantization scaling factor wS corresponding to each weight part is extracted along the K dimension, and the shape of the weight quantization scaling factor wS is (Ks, 1, N). , 1, N).

[0070] For example, the shape of the weight quantization scaling factor wS is (2, 1, 2), and its value is [[[0.32, 0.40]], [[0.32, 0.40]]].

[0071] In some embodiments, in the step 120, obtaining the noise suppression coefficient related to the optical computing system includes:

[0072] S121, obtaining an activation noise suppression coefficient xNSF.

[0073] S122, obtaining a weight noise suppression coefficient wNSF.

[0074] Exemplarily, refer to FIG. 3, which is a flow chart of the calculation of the activation noise suppression coefficient according to an embodiment of the present application. The step S121 includes S1211 to S1214.

[0075] S1211, determining the maximum value maxX and the minimum value minX of the quantized activation matrix qX along the Ks dimension, the shape of the quantized activation matrix qX being (B x M, Ks). , B x M, Ks).

[0076] That is, the elements of the quantized activation matrix qX are traversed in a particular Ks dimension to find the maximum (maxX) and minimum (minX) values in each Ks group.

[0077] For example, for [[ [3, 1], [3, -3] ], [ [-3, 0], [1, 0] ] ], the shapes of both maxX and minX are (2, 2, 1). Specifically, for [3, 1] in it, the minX is [1] and the max X is [3], so for [[ [3, 1], [-3, 0] ], [[3, -3], [1, 0] ] ], the corresponding minX is [[ [1], [-3] ], [ [-3], [0] ] ] and the maxX is [[ [3], [3] ], [[0], [1] ] ].

[0078] S1212, according to the minimum value minE and the maximum value maxE of the conversion range of the digital-to-analog converter, determine the noise suppression base NFS.

[0079] Specifically, please refer to FIG. 4, which is a schematic diagram of the optical computing architecture provided by the embodiment of the present application. The first data and the second data are in the form of digital electrical signals. The first data and the second data can be related data of activation matrix and weight matrix in a neural network model, respectively. The function of the digital-to-analog converter (DAC) is to convert digital electrical signals into analog electrical signals, and to generate optical signals carrying data (input vector) based on the analog electrical signals, i.e. electro-optical conversion, for example, modulating optical signals based on analog electrical signals. The optical computing unit can include a modulator to realize the modulation function. The optical computing unit can process related optical signals, and then perform photoelectric conversion, for example, photoelectric conversion into analog electrical signals. The DAC has a minimum value (minE) and a maximum value (maxE) range when converting. The range is determined by the characteristics of the DAC device itself. For example, different specifications of DAC chips have different minimum and maximum signal values that can be converted. Taking a DAC capable of encoding int4 data as an example, the int4 data type represents a signed integer represented by 4-bit binary numbers. In this case, minE is -8 and maxE is 7.

[0080] Exemplarily, the maximum of the absolute value of the minimum value minE and the absolute value of the maximum value maxE can be taken as the noise suppression base NSF.

[0081] That is, the calculation of the NFS (Noise Suppression Factor) is obtained by taking the maximum of the absolute value of minE and the absolute value of maxE, i.e., NFS = max(abs(minE), abs(maxE)), where abs() is the absolute value function and max() is the maximum function. For example, in the previous example (minE = -8, maxE = 7), NFS = 8. The NFS is related to minE and maxE, and can also be referred to as the noise suppression factor.

[0082] S1213, determine a first parameter nNSF according to the minimum value minE and the minimum value minX, and determine a second parameter pNSF according to the maximum value maxE and the maximum value maxX.

[0083] (1) Determine the first parameter nNSF:

[0084] For example, if the minimum value minE is less than 0 and the minimum value minX is also less than 0, the first parameter nNSF is equal to the result of minE divided by minX and rounded down; otherwise, the first parameter nNSF is equal to the noise suppression factor NSF.

[0085] Specifically, when calculating the first parameter nNSF, when the condition minE < 0 and minX < 0 is met, the calculation of nNSF is performed according to the formula , where represents rounding down.

[0086] For example, if minE is -4 and minX is -2, then Here, minE is usually less than 0, and if minE is greater than or equal to 0, the calculation of this branch is not performed.

[0087] At the same time, the shape of minX and the shape of nNSF are the same, and the positions are one-to-one corresponding. That is, minX is a matrix (shape [2, 2, 1]), and for each position of minX that is less than 0, the corresponding position of nNSF is calculated according to the above formula to obtain the corresponding value.

[0088] For example, nNSF = [[[-], [2]], [[2], [-]]], where - represents no value, which may be because minX does not meet the condition of being less than 0 at this position, so no value is calculated according to , and the no value situation occurs.

[0089] If the condition minE < 0 and minX < 0 is not met, then nNSF = NSF, and NFS = max(abs(minE), abs(maxE)).

[0090] For example, the final result of the first parameter nNSF is [[[8], [2]], [[2], [8]]], which has a shape of (2, 2, 1), a three-dimensional matrix, which is the same as the shape of minX before, which means that each element in nNSF is one-to-one corresponding to the element in minX in position.

[0091] (2) Determine the second parameter pNSF:

[0092] Exemplarily, if the maximum value maxE is greater than 0 and the maximum value maxX is also greater than 0, the second parameter pNSF is equal to the result of maxE divided by maxX and rounded down; otherwise, the second parameter pNSF is equal to the noise suppression base NSF.

[0093] Specifically, when calculating the second parameter pNSF, when the conditions maxE>0 and maxX>0 are met, the calculation is performed according to the formula .

[0094] For example, if maxE is 4 and maxX is 2, then Here, maxE is usually greater than 0, and if maxE is less than or equal to 0, the calculation of this branch is not performed.

[0095] For the condition that maxX>0, it means that the value of pNSF corresponding to the position greater than 0 in maxX is calculated according to the formula . For example, in the given example, pNSF = [[[2], [2]], [[-], [7]]], and if maxX does not meet the condition of being greater than 0 at a certain position, - may appear, indicating that there is no value.

[0096] If the conditions maxE>0 and maxX>0 are not met, then pNSF=NSF.

[0097] For example, the final result of the second parameter pNSF is [[[2], [2]], [[8], [7]]], which is also a three-dimensional matrix with a shape of (2, 2, 1).

[0098] S1214, according to the first parameter nNSF and the second parameter pNSF, determine the activated noise suppression coefficient xNSF, which has a shape of .

[0099] Exemplarily, the minimum value of the first parameter nNSF and the second parameter pNSF is taken as the activated noise suppression coefficient xNSF.

[0100] That is, xNSF=min(nNSF, pNSF), min() is the minimum value function. The shape of xNSF is ( , B x M, 1), represents a rounding up symbol.

[0101] For example, nNSF = [[[8], [2]], [[2], [8]]], pNSF = [[[2], [2]], [[8], [7]]]. For the calculation of xNSF, for example, at the element position of the first row and the first column of xNSF, the value in nNSF is 8, and the value in pNSF is 2, and the minimum value is taken by the min() function, so the value of xNSF at this position is 2. In this way, the elements at each position are calculated, and finally xNSF = [[[2], [2]], [[2], [7]]] is obtained, and the shape is (2, 2, 1).

[0102] Exemplarily, refer to FIG. 5, which is a flow chart of the calculation of the weight noise suppression coefficient provided by the embodiment of the present application. The obtaining of the weight noise suppression coefficient wNSF in the above step S122 includes S1221 to S1224.

[0103] S1221, determining the maximum value maxW and the minimum value minW of the quantized weight matrix qW in the Ks dimension, the shape of the quantized weight matrix qW being (K, N). , K, N).

[0104] That is, the elements of the quantized weight matrix qW are traversed in a specific Ks dimension, and the maximum value and the minimum value in each Ks group are found.

[0105] For example, the shape of maxW and minW is both (2, 1, 1), the value of maxW is [[[1, 3]], [[1, 3]]], and the value of minW is [[[0, 0]], [[-3, 1]]].

[0106] S1222, determining the noise suppression base NSF according to the minimum value minE and the maximum value maxE of the conversion range of the digital-to-analog converter.

[0107] Specifically, refer to FIG. 4 again, the weight vector is also in digital form and needs to be converted into an analog signal by a DAC. The conversion process is similar to that of the input vector. The DAC has a minimum value (minE) and a maximum value (maxE) range when converting. The range is determined by the characteristics of the DAC device itself.

[0108] Exemplarily, the maximum of the absolute value of the minimum value minE and the absolute value of the maximum value maxE can be taken as the noise suppression base NSF.

[0109] That is, the calculation of the NFS (Noise Suppression Factor) is obtained by taking the maximum of the absolute value of minE and the absolute value of maxE, i.e., NFS = max(abs(minE), abs(maxE)), where abs() is the absolute value function and max() is the maximum function. For example, in the previous example (minE = -8 and maxE = 7), NFS = 8.

[0110] S1223, determining a first parameter nNSF' according to the minimum value minE and the minimum value minW, and determining a second parameter pNSF' according to the maximum value maxE and the maximum value maxW.

[0111] (1) Determining the first parameter nNSF':

[0112] For example, if the minimum value minE is less than 0 and the minimum value minW is also less than 0, the first parameter nNSF' is equal to the result of minE divided by minW and then rounded down; otherwise, the first parameter nNSF' is equal to the noise suppression base NSF.

[0113] Specifically, when calculating the first parameter nNSF', when minE < 0 and minW < 0 are satisfied, the calculation of nNSF' is performed according to the formula , where represents rounding down. If this condition is not satisfied (i.e., minE ≥ 0 or minW ≥ 0), then nNSF' = NSF.

[0114] For example, the first parameter nNSF' = [[[8, 8]], [[2, 8] ].

[0115] (2) Determining the second parameter pNSF':

[0116] For example, if the maximum value maxE is greater than 0 and the maximum value maxW is also greater than 0, the second parameter pNSF' is equal to the result of maxE divided by maxW and then rounded down; otherwise, the second parameter pNSF' is equal to the noise suppression base NSF.

[0117] Specifically, when calculating the second parameter pNSF', when maxE > 0 and maxW > 0 are satisfied, pNSF' is calculated according to the formula . If this condition is not satisfied (i.e., maxE < 0 or maxW ≤ 0), then pNSF' = NSF.

[0118] For example, the finally obtained second parameter pNSF' = [[[7, 2]], [[7, 2] ].

[0119] S1224, determining a weight noise suppression coefficient wNSF according to the first parameter nNSF' and the second parameter pNSF', the shape of wNSF is (N, 1, N).

[0120] Exemplarily, taking the minimum value of the first parameter nNSF' and the second parameter pNSF' as the weight noise suppression coefficient wNSF.

[0121] That is, wNSF = min(nNSF', pNSF'), min() is the minimum value function. The shape of wNSF is [N, 1, N]. is the upward rounding symbol, wNSF is a three-dimensional matrix, the size of the first dimension is the size of the second dimension is 1, and the size of the third dimension is N.

[0122] For example, nNSF' = [[[8, 8]], [[2, 8]]], pNSF' = [[[7, 2]], [[7, 2]]]. Finally, wNSF = [[[7, 2]], [[2, 2]]], the shape is (2, 1, 2). wNSF is calculated according to the rule of taking the minimum value of nNSF' and pNSF', for example, the value of nNSF' at a certain position is 8, and the value of pNSF' at the corresponding position is 7, then the value of wNSF at the position is 7.

[0123] Please refer to FIG. 6, which is a schematic diagram of the optimization method of the neural network model in the optical calculation provided by the embodiment of the present application. Exemplarily, in the step 130, the initial quantization scaling coefficient is adjusted according to the noise suppression coefficient to obtain the adjusted quantization scaling coefficient, which includes:

[0124] S131, taking the ratio of the initial quantization scaling coefficient and the noise suppression coefficient as the adjusted quantization scaling coefficient. Specifically, it includes:

[0125] S1311, taking the ratio of the activation quantization scaling coefficient xS and the activation noise suppression coefficient xNSF as the adjusted activation quantization scaling coefficient xS'.

[0126] For example, xS = [[[0.52], [0.24]], [[0.52], [0.24]]], the shape is (2, 2, 1). xNSF = [[[2], [2]], [[2], [7]]], the shape is (2, 2, 1). Since the shape of xS and xNSF is (2, 2, 1), the corresponding elements can be divided to calculate xS':

[0127] ① For the first row and first column elements of xS and the first row and first column elements of xNSF: ​​

[0128] The first row first column element of xS is [0.52], and the first row first column element of xNSF is [2]. Calculate xS' = xS / xNSF = 0.52 / 2 = 0.26.

[0129] ②For the first row second column element of xS and the first row second column element of xNSF:

[0130] The first row second column element of xS is [0.24], and the first row second column element of xNSF is [2]. Calculate xS' = xS / xNSF = 0.24 / 2 = 0.12.

[0131] ③For the second row first column element of xS and the second row first column element of xNSF:

[0132] The second row first column element of xS is [0.52], and the second row first column element of xNSF is [[2]]. Calculate xS' = xS / xNSF = 0.52 / 2 = 0.26.

[0133] ④For the second row second column element of xS and the second row second column element of xNSF:

[0134] The second row second column element of xS is [[0.24]], and the second row second column element of xNSF is [[7]]. Calculate xS' = xS / xNSF = 0.24 / 7 ≈ 0.03 (rounding to two decimal places).

[0135] Therefore, xS' = [[[0.26], [0.12]], [[0.26], [0.03]]], with a shape of (2, 2, 1).

[0136] S1312, the ratio of the weight quantization scaling coefficient wS and the weight noise suppression coefficient wNSF is taken as the adjusted weight quantization scaling coefficient wS'.

[0137] For example, wS = [[[0.32, 0.40]], [[0.32, 0.40]]], with a shape of (2, 1, 2). wNSF = [[[7, 2]], [[2, 2]]], with a shape of (2, 1, 2). Since the shapes of wS and wNSF are both (2, 1, 2), the corresponding elements can be divided to calculate wS':

[0138] ①For the first element of the first row first column of wS and the first element of the first row first column of wNSF:

[0139] The first element of the first row first column of wS is 0.32, and the first element of the first row first column of wNSF is 7. Calculate wS' = 0.32 / 7 ≈ 0.05 (rounding to two decimal places).

[0140] ii) for the second element of the first row of the first column of wS and the second element of the first row of the first column of wNSF:

[0141] The second element of the first row of the first column of wS is 0.40 and the second element of the first row of the first column of wNSF is 2. Calculate wS' = 0.40 / 2 = 0.20.

[0142] iii) for the first element of the second row of the first column of wS and the first element of the second row of the first column of wNSF:

[0143] The first element of the second row of the first column of wS is 0.32 and the first element of the second row of the first column of wNSF is 2. Calculate wS' = 0.32 / 2 = 0.16.

[0144] iv) for the second element of the second row of the first column of wS and the second element of the second row of the first column of wNSF:

[0145] The second element of the second row of the first column of wS is 0.40 and the second element of the second row of the first column of wNSF is 2. Calculate wS' = 0.40 / 2 = 0.20.

[0146] Therefore, wS' = [[[0.05, 0.20]], [[0.16, 0.20]]], which has a shape of (2, 1, 2).

[0147] In some embodiments, the method for optimizing a neural network model in the optical computing further comprises:

[0148] S150, performing a matrix multiplication calculation, specifically comprising S151 and S152:

[0149] S151, quantizing the activation matrix X according to the adjusted activation quantization scaling factor xS', to obtain a quantized activation matrix qX'; and further adjusting the weight matrix W according to the adjusted weight quantization scaling factor wS', to obtain a quantized weight matrix qW'.

[0150] S152, performing a matrix multiplication operation on the quantized activation matrix qX' and the quantized weight matrix qW', to obtain a matrix multiplication calculation result qO, which has a shape of (B x M x N).

[0151] ​For example, qX' = [[[3, 1], [3, -3]], [[-3, 0], [1, 0]]], shape (2, 2, 2); qW' = [[[0, 0], [1, 3]], [[-3, 3], [1, 1]]], shape (2, 2, 2). Then qO = qX' * qW' = [[[1, 3], [-3, -9]], [[9, -9], [-3, 3]]], shape (2, 2, 2).

[0152] In this way, the neural network model in optical computing can be optimized by optimizing the quantization process.

[0153] In some embodiments, the application also provides a dequantization method (step) of a neural network model in optical computing, which comprises dequantization processing by using the optimization method of the neural network model in optical computing according to any of the above embodiments.

[0154] Exemplarily, the dequantization method of the neural network model in optical computing comprises:

[0155] S160, performing corresponding element multiplication with broadcast on the adjusted activation quantization scaling factor xS' and the adjusted weight quantization scaling factor wS' to obtain a dequantization scaling factor dS, i.e. dS = xS'. wS', and the shape of dS is (B, M, N).

[0156] The dequantization scaling factor dS is obtained by matrix multiplication of xS' and wS', i.e. dS = xS' wS'. Where. represents corresponding point multiplication with broadcast, and the shape of the dequantization matrix is (B, M, N).

[0157] In some embodiments, the dequantization of the neural network model in optical computing further comprises:

[0158] S170, multiplying the dequantization scaling factor dS and the matrix multiplication result qO to obtain a dequantization calculation result O.

[0159] S180, performing cumulative summation on the dequantization calculation result O according to the specified dimension to obtain a summation result S after dequantization, and the shape of the summation result S is (1, B x M, N).

[0160] Specifically, the cumulative summation on the dequantization calculation result O is performed according to the 0 dimension, i.e. the summation result S after dequantization = O.sum(dim=0), which means that the elements of the O matrix in the first dimension are summed. The shape of the output after summation becomes [1, B x M, N].

[0161] S190, performing a shape conversion operation on the summation result S after dequantization to obtain a final result R of matrix multiplication, and the shape of the final result R of matrix multiplication is (B, M, N).​​

[0162] Specifically, the shape of the dequantized summation result S is finally converted from [1, B x M, N] to [B, M, N] through a reshape operation.

[0163] The neural network model optimization device in optical computing provided in the present application is described below, and the neural network model optimization device described below can be mutually referred to the optimization method of the neural network model in optical computing described above.

[0164] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, and the computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to ensure that a computer device (which can be a personal computer, a server, or a network device, etc.) executes the methods described in each embodiment or some parts of the embodiments.

[0165] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for optimizing a neural network model in optical computing, wherein the method comprises: obtaining initial quantization scaling factors; obtaining a noise suppression factor related to an optical computing system; adjusting the initial quantization scaling factors according to the noise suppression factor to obtain adjusted quantization scaling factors; quantizing a neural network model using the adjusted quantization scaling factors.

2. The method for optimizing a neural network model in optical computing according to claim 1, wherein the adjusting the initial quantization scaling factors according to the noise suppression factor to obtain adjusted quantization scaling factors comprises: taking a ratio of the initial quantization scaling factors to the noise suppression factor as the adjusted quantization scaling factors.

3. The method for optimizing a neural network model in optical computing according to claim 1, wherein the obtaining initial quantization scaling factors comprises: determining the initial quantization scaling factors according to a maximum value and a minimum value of floating-point numbers corresponding to data to be quantized.

4. The method for optimizing a neural network model in optical computing according to claim 1, wherein the method further comprises: the quantizing the neural network model comprises quantizing at least one of an activation matrix and a weight matrix.

5. The method for optimizing a neural network model in optical computing according to claim 4, wherein comprises obtaining an activation matrix X to be quantized, the shape of the activation matrix X being (B, M, K), wherein B represents a batch size, M represents a row of the matrix, and K represents a column of the matrix.

6. The method for optimizing a neural network model in optical computing according to claim 5, wherein the method further comprises: performing a shape conversion operation on the activation matrix X, comprising: A shape conversion operation is performed on the activation matrix X to convert its shape from (B, M, K) to wherein Ks represents the number of summation objects in the dot product summation unit of the optical computing hardware, represents rounding up; if K cannot be divided by Ks, the elements of the insufficient part are padded with 0. A matrix transpose operation is performed on the activation matrix X to change its shape from converted to 7. The method for optimizing a neural network model in optical computing according to claim 5, wherein the method further comprises: extracting an initial quantized scaling coefficient xS corresponding to each activated part along the Ks dimension, and taking the representation layer shape of the activated initial quantized scaling coefficient xS as a matrix of.

8. The method for optimizing a neural network model in optical computing according to claim 4, wherein comprises obtaining a weight matrix W to be quantized: the shape of the weight matrix W is (K, N), wherein K represents a row of the matrix, and N represents a column of the matrix.

9. The method for optimizing a neural network model in optical computing according to claim 8, wherein the method further comprises: The shape conversion operation is performed on the weight matrix W to convert its shape from (K, N) to wherein Ks represents the number of summation objects in the dot product summation unit of the optical computing hardware, represents rounding up; if K cannot be divided by Ks, the elements of the insufficient part are padded with 0.

10. The method for optimizing a neural network model in optical computing according to claim 9, wherein the method further comprises: extracting an initial quantization scaling coefficient wS corresponding to each weight part along the K dimension, and representing the initial quantization scaling coefficient wS of the weight as a shape of a matrix of.

11. The method for optimizing a neural network model in optical computing according to claim 1, wherein the obtaining a noise suppression factor related to an optical computing system comprises: determining a noise suppression base number NFS according to a minimum value minE and a maximum value maxE of a conversion range of a digital-to-analog converter in the optical computing system; obtaining the maximum value max X and the minimum value min X of the activation matrix X in the dimension of Ks, wherein the activation matrix X has a shape of determining a first parameter nNSF according to the minimum value minE and the minimum value minX, and determining a second parameter pNSF according to the maximum value maxE and the maximum value maxX; According to the first parameter nNSF and the second parameter pNSF, a noise suppression coefficient xNSF is determined, which is shaped as 12. The method of Claim 1, wherein the obtaining a noise suppression factor related to the optical computing system comprises: obtaining a maximum value maxW and a minimum value minW of the weight matrix W in the Ks dimension, wherein the weight matrix W has a shape of determining a noise suppression base number NFS according to a minimum value minE and a maximum value maxE of a conversion range of a digital-to-analog converter; determining a first parameter nNSF' according to the minimum value minE and a minimum value minW, and determining a second parameter pNSF' according to the maximum value maxE and a maximum value maxW; A noise suppression coefficient wNSF of the weight is determined according to the first parameter nNSF' and the second parameter pNSF', which is shaped as 13. The method of Claim 10 or 11, wherein a maximum value between an absolute value of the minimum value minE and an absolute value of the maximum value maxE is taken as the noise suppression base number NFS.

14. The method of Claim 10 or 11, wherein, if the minimum value minE is less than 0 and the minimum value minX is also less than 0, the first parameter nNSF is equal to a result of minE divided by minX rounded down; otherwise, the first parameter nNSF is equal to the noise suppression base number NFS; if the minimum value minE is less than 0 and the minimum value minW is also less than 0, the first parameter nNSF' is equal to a result of minE divided by minW rounded down; otherwise, the first parameter nNSF' is equal to the noise suppression base number NFS.

15. The method of Claim 10 or 11, wherein, if the maximum value maxE is greater than 0 and the maximum value maxX is also greater than 0, the second parameter pNSF is equal to a result of maxE divided by maxX rounded down; otherwise, the second parameter pNSF is equal to the noise suppression base number NFS; if the maximum value maxE is greater than 0 and the maximum value maxW is also greater than 0, the second parameter pNSF' is equal to a result of maxE divided by maxW rounded down; otherwise, the second parameter pNSF' is equal to the noise suppression base number NFS.

16. The method of Claim 10 or 11, wherein, a minimum value between the first parameter nNSF and the second parameter pNSF is taken as a noise suppression factor for activations xNSF; a minimum value between the first parameter nNSF' and the second parameter pNSF' is taken as a noise suppression factor for weights wNSF.

17. The method of Claim 2, wherein the taking a ratio of the initial quantized scaling factor and the noise suppression factor as the adjusted quantized scaling factor comprises: taking a ratio of an initial quantized scaling factor for activations xS and a noise suppression factor for activations xNSF as an adjusted quantized scaling factor for activations xS'; taking a ratio of an initial quantized scaling factor for weights wS and a noise suppression factor for weights wNSF as an adjusted quantized scaling factor for weights wS'.

18. The method of Claim 17, wherein the quantizing the neural network model using the adjusted quantized scaling factor further comprises: According to the adjusted activation quantization scaling factor xS', the activation matrix X to be quantized is quantized to obtain a quantized activation matrix qX'; and according to the adjusted weight quantization scaling factor wS', the weight matrix qW is quantized to obtain a quantized weight matrix qW'; For the quantized activation matrix qX' and the quantized weight matrix qW', a matrix multiplication operation is performed to obtain a matrix multiplication calculation result qO, which has a shape of 19. The optimization method of a neural network model in optical computing according to claim 18, wherein the method further comprises: The adjusted activation quantization scaling factor xS' and the adjusted weight quantization scaling factor wS' are performed with a corresponding element multiplication with broadcast, i.e., dS = xS' · wS', resulting in a dequantization scaling factor dS, which has the shape multiplying the dequantization scaling factor dS and the matrix multiplication calculation result qO to obtain a dequantized result O.

20. The method according to claim 19, wherein the method further comprises: performing cumulative summation on the dequantization calculation result O according to the specified dimensions to obtain a dequantized summation result S, which has a shape of (1, BxM, N); performing a shape conversion operation on the dequantized summation result S to obtain a matrix multiplication final result R, which has a shape of (B, M, N).

Images