Vector approximate updating method of sparse neural network and application
Through the vector approximate update method of sparse neural network, the high cost problem caused by traditional weight update algorithm is solved, efficient sparse training is achieved, time and energy consumption overhead is reduced, and high-precision update effect is maintained.
Patent Information
- Application Number
- CN202410956818.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-11-08
AI Technical Summary
The traditional weight update algorithm was used during the training of sparse neural networks, which resulted in high time and energy consumption costs, and required external indexing of sparse information many times. The failure to effectively utilize the advantages of sparseness, which concealed the theory of sparse neural networks. Advantages.
Using the vector approximate update method of sparse neural network, by obtaining the precise gradient information of the weight parameters to be updated, building a matrix structure, building operation matrix variables and gradient matrix variables, fitting and solving the optimal operation matrix, performing weight parameter updates, and reducing update operations Quantity and condition types, avoid external indexing.
The time and energy consumption overhead during sparse training are significantly reduced, and the precise update effect of updating one unit is maintained. The mean square error significantly decreases when the sparse rate is increased, and the accuracy is close to the traditional update algorithm.
Smart Images

Figure CN118917368A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of neural network technology, and in particular to a vector approximation updating method and application of a sparse neural network. Background Art
[0002] With the rapid development of artificial intelligence (AI), the scale of neural networks has also grown exponentially, and model compression technology represented by sparse neural networks has received unprecedented attention. The idea of sparse neural networks is to prune the weights with smaller values in dense networks so that they do not participate in training and reasoning, thereby directly improving computing speed and power consumption.
[0003] However, the ideal advantages of sparse neural networks have not been realized. One reason is that the traditional weight update algorithm developed from dense networks is currently used to train sparse neural networks, such as unit-by-unit updates, row-column parallel updates, etc., with many updates and many types of update conditions. In addition, multiple external indexes of sparse information are required, and the time and energy costs of updates are high. As a result, not only the advantages of sparsity cannot be utilized, but also more additional overhead is caused, which obscures the theoretical advantages of sparse neural networks. Summary of the invention
[0004] The purpose of the present application is to provide a vector approximation update method and application of a sparse neural network, so as to solve the problem that the sparse neural network training in the prior art adopts the traditional weight update algorithm developed from the dense network, the time cost and energy consumption cost of the update are large, and multiple external indexes of the sparse information are required, which not only fails to utilize the advantages of sparsity, but also causes more additional overhead, obscuring the theoretical advantages of the sparse neural network.
[0005] To achieve the above-mentioned purpose, the first aspect of the present application provides a vector approximation updating method of a sparse neural network, comprising:
[0006] Obtain the precise gradient parameters of the weight parameters to be updated in the neural network and obtain precise gradient information;
[0007] Based on the precise gradient information, a matrix structure of a weight matrix is obtained, and the same update pulse is constructed for multiple weight parameters in the same row and multiple weight parameters in the same column in the matrix structure to obtain multiple row vectors and multiple column vectors;
[0008] Constructing an operation matrix variable based on the multiple row vectors and the multiple column vectors;
[0009] Based on the operation matrix variable, construct a gradient matrix variable, each gradient parameter in the gradient matrix variable corresponds to a weight parameter in the weight matrix;
[0010] Taking the accurate gradient matrix as the target of the gradient matrix variable, fitting and solving the operation matrix variable to obtain the optimal operation matrix;
[0011] Based on the optimal operation matrix, the weight parameters of the neural network are updated.
[0012] In one or more embodiments, the step of obtaining the precise gradient parameters of the weight parameters to be updated in the neural network is specifically as follows:
[0013] The aligned label values are compared with the output values of the neural network, and the precise gradient parameters of the weight parameters to be updated in the neural network are obtained based on the back propagation algorithm.
[0014] In one or more embodiments, the step of constructing a gradient matrix variable based on the operation matrix variable is specifically:
[0015] Constructing an auxiliary matrix so that the sum of the products of the corresponding bits of any row or column of the operation matrix variable and any column or row of the auxiliary matrix is a gradient parameter, and the gradient parameter is the sum of the row vector and the column vector corresponding to a weight parameter in the weight matrix;
[0016] Based on the auxiliary matrix, a functional relationship A*OP=ΔW between the operation matrix variable and the gradient matrix variable is obtained, wherein matrix A is the auxiliary matrix, matrix OP is the operation matrix variable, and matrix ΔW is the gradient matrix variable.
[0017] In one or more embodiments, the weight matrix is an M*N matrix, the operation matrix variable is an (M+N)*1 matrix, the gradient matrix variable is an MN*1 matrix, and the auxiliary matrix is an MN*(M+N) matrix.
[0018] In one or more embodiments, the step of taking the accurate gradient matrix as the target of the gradient matrix variable, fitting and solving the operation matrix variable, and obtaining the optimal operation matrix is specifically:
[0019] Substituting the precise gradient information into the gradient matrix variable to obtain a precise gradient matrix;
[0020] Solving the functional relationship based on the least squares method, we get the equation: OP = (A T A) -1 *A T *ΔW;
[0021] Substituting the precise gradient matrix and the auxiliary matrix into the equation, the optimal operation matrix is calculated.
[0022] In one or more embodiments, it further includes:
[0023] Based on the optimal operation matrix, an approximate gradient matrix is obtained;
[0024] The mean square error between the approximate gradient matrix and the precise gradient matrix is calculated as a performance evaluation indicator.
[0025] To achieve the above-mentioned purpose, the second aspect of the present application provides a vector approximation updating device for a sparse neural network, comprising:
[0026] An acquisition module is used to obtain the precise gradient parameters of the weight parameters to be updated in the neural network and obtain precise gradient information;
[0027] A vectorization module, used for obtaining a matrix structure of a weight matrix based on the precise gradient information, and constructing the same update pulse for multiple weight parameters in the same row and multiple weight parameters in the same column in the matrix structure to obtain multiple row vectors and multiple column vectors;
[0028] A first construction module, configured to construct an operation matrix variable based on the plurality of row vectors and the plurality of column vectors;
[0029] A second construction module is used to construct a gradient matrix variable based on the operation matrix variable, each gradient parameter in the gradient matrix variable corresponds to a weight parameter in the weight matrix;
[0030] A fitting module is used to fit and solve the operation matrix variables with an accurate gradient matrix as the target of the gradient matrix variables to obtain an optimal operation matrix;
[0031] An updating module is used to update the weight parameters of the weight matrix based on the optimal operation matrix.
[0032] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including:
[0033] at least one processor; and
[0034] A memory storing instructions, which, when executed by the at least one processor, enables the at least one processor to execute the vector approximation updating method as described in any one of the above embodiments.
[0035] To achieve the above objectives, the fourth aspect of the present application provides a machine-readable storage medium storing executable instructions, which, when executed, enable the machine to execute the vector approximation update method as described in any of the above embodiments.
[0036] Different from the prior art, the beneficial effects of this application are:
[0037] When the vector approximation update method of the present application updates the weight matrix M*N of the sparse neural network, the number of update operations is M+1 or N+1, the types of update conditions are M+N, and there is no need to perform external indexing of sparse information throughout the process. Compared with the traditional one-by-one update algorithm and row-column parallel update algorithm, the time and energy consumption in the sparse training process are greatly reduced;
[0038] The vector approximation update method of the present application has an update accuracy and training effect comparable to that of the traditional precise update algorithm of unit-by-unit update in the process of updating sparse neural networks. Even at a relatively low sparsity, the mean square error between the approximate gradient matrix and the target matrix is much less than 1 state. When the sparsity rate increases from 52.3% to 74.2%, the mean square error drops from 0.24 to 0.06, which is almost negligible. Compared with the traditional unit-by-unit update algorithm, the method of the present application has an accuracy slightly lower than that of the traditional unit-by-unit update algorithm at low sparsity rates (20%), and the accuracy of the method of the present application is comparable to that of the unit-by-unit update algorithm at other sparsity rates. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art description are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0040] Figure 1 It is a flowchart of an implementation method of a vector approximation updating method of a sparse neural network of the present application;
[0041] Figure 2 This is a schematic diagram of the vectorized approximate update operation using a 2T storage-computation integrated unit in this application;
[0042] Figure 3 yes Figure 1 A schematic diagram of a flow chart of an implementation method corresponding to S400;
[0043] Figure 4 This is the relationship between the mean square error and the sparsity rate of the VAU algorithm of this application;
[0044] Figure 5 This is a data chart comparing the sparse training accuracy of the VAU algorithm and the Cell-by-cell algorithm of this application;
[0045] Figure 6It is a structural schematic diagram of an implementation method of a vector approximation updating device of a sparse neural network of the present application;
[0046] Figure 7 It is a schematic diagram of the hardware structure of the electronic device of this application. DETAILED DESCRIPTION
[0047] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of this application.
[0048] Inspired by the sparsity of the human brain, the idea of sparse neural networks is to prune the weights with smaller values in dense networks so that they do not participate in training and reasoning, thereby directly improving computing speed and power consumption. For medium and large networks, the sparsity rate is often as high as 90%, which means an order of magnitude performance improvement.
[0049] However, the ideal advantages of sparse neural networks have not been realized. One reason is that modern AI hardware relies on indexing of off-chip memory to complete pruning. Ideal fine-grained, unstructured pruning has excessive indexing overhead, which directly obscures the sparse advantage, while structured, coarse-grained pruning reduces indexing overhead but sacrifices accuracy - this is the "Accuracy-Granularity dilemma" faced by sparse neural networks, and there is still no good solution.
[0050] On the other hand, for sparse neural networks, joint optimization and algorithm innovation for new hardware are still lacking. Traditional weight update algorithms developed from dense networks, such as unit-by-unit update and row-column parallel update, not only fail to take advantage of sparsity, but also cause more additional overhead, obscuring the theoretical advantages of sparse neural networks.
[0051] Specifically, in the cell-by-cell method, for a dense neural network with M rows and N columns, using cell-by-cell update, the number of updates required is M*N times, and the types of update conditions required are M*N. For a sparse network of the same size, the number of updates is M*N*(1-Sparsity), the types of update conditions are M*N*(1-Sparsity), and M*N indexes for sparsity information are required. This update method is the most accurate, but it also takes the most time and energy;
[0052] In the row-column parallel update method, for a dense neural network with M rows and N columns, using column-by-column update, the number of updates required is N times, and the types of update conditions required are M*N; for a sparse network of the same size, although the number of updates is N times, the types of update conditions required are M*N, and in addition, N indexes of the sparsity information of the entire column are required. Although this update method is parallel and reduces the update time, it still requires the periphery to provide multiple update conditions and multiple external indexes of sparse information.
[0053] Therefore, the current sparse neural network update method has high time and energy costs for updating, and requires multiple external indexes of sparse information, which not only fails to take advantage of sparsity, but also causes more additional overhead, obscuring the theoretical advantages of sparse neural networks.
[0054] In order to solve the above problems, the applicant has developed a new vector approximate update algorithm (VAU) for sparse neural networks. On the one hand, this update algorithm can significantly reduce the number of weight updates, and there is no need for external indexing of sparse information throughout the process, which greatly reduces the time and energy consumption in the sparse training process. In addition, the algorithm has update accuracy and training effect comparable to unit-by-unit updates in sparse networks.
[0055] Specifically, see Figure 1 , Figure 1 It is a flowchart of an implementation method of a vector approximation update method of a sparse neural network of the present application.
[0056] like Figure 1 As shown, the method includes:
[0057] S100, obtaining precise gradient parameters of weight parameters to be updated in the neural network, and obtaining precise gradient information.
[0058] Among them, the precise gradient parameters can be obtained through the back propagation algorithm.
[0059] Specifically, the registered label value and the output value of the neural network can be compared to obtain the output error of the neural network.
[0060] The output error is back-propagated in the neural network, and the error of each weight parameter to be updated in the neural network, that is, the precise gradient parameter, can be obtained.
[0061] S200, based on the precise gradient information, obtaining the matrix structure of the weight matrix, and constructing the same update pulse for multiple weight parameters in the same row and multiple weight parameters in the same column in the matrix structure to obtain multiple row vectors and multiple column vectors.
[0062] Since the precise gradient information includes the precise gradient parameters of all weight parameters to be updated, the number of weight parameters can be obtained based on the precise gradient information, that is, the matrix structure of the weight matrix can be obtained.
[0063] For the weight matrix, the traditional update method is to construct a gradient for each weight parameter. For M*N weight parameters, the traditional update method constructs M*N gradients.
[0064] The method of the present application adopts a gradient vector solution to replace the traditional gradient solution. Specifically, a consistent update pulse can be applied to all weight parameters of the mth row, which can be written as a vector OP(m,), thereby obtaining m row vectors, which can be expressed as:
[0065]
[0066] Correspondingly, a consistent update pulse can be applied to all weight parameters of the nth column, which can be written as a vector OP(,n), thereby obtaining n column vectors, which can be expressed as:
[0067] OP Col, =[OP(,N) … OP(,2) OP(,1)].
[0068] Based on the above steps, M+N gradient vectors are obtained. The M+N gradient vectors are all operations that need to be determined by the algorithm of this application. Since the storage and computing integrated array shares row and column lines in a certain direction, the total number of operations can be further reduced to M+1 or N+1.
[0069] For example, taking a 2T storage and computing integrated unit as an example, see Figure 2 , Figure 2 This is a schematic diagram of the vectorized approximate update operation using a 2T storage and computing integrated unit in this application. Figure 2As shown, each unit in the storage-computing integrated unit includes two ferroelectric transistors, one of which is used to control the switch of the other transistor, and the word lines of the units in the same row are collinear. Since the word lines are horizontal, M rows can be selected at the same time, and M vectors of row operations can be applied at the same time, requiring a total of N+1 operations.
[0070] It should be noted that the vectorized approximate update method of the present application is not limited to the application scenario of the above-mentioned 2T storage-computing integrated unit, but can also be applied to any other device with integrated storage and computing capabilities, for example, it can also be applied to a storage device with a three-terminal structure. Exemplarily, each storage-computing integrated unit can include a storage device with a three-terminal structure and a memory such as RRAM with a two-terminal structure, both of which can achieve the effect of the present embodiment.
[0071] It can be understood that, compared with the traditional one-by-one updating method, the above-mentioned vectorization-based method can reduce power consumption and delay by M times or N times.
[0072] S300. Construct an operation matrix variable based on multiple row vectors and multiple column vectors.
[0073] For subsequent matrix processing, the multiple row vectors and the multiple column vectors obtained in the above S200 may be unified into a single matrix to obtain an operation matrix variable.
[0074] Taking the row vector and column vector in S200 as an example, the operation matrix can be:
[0075]
[0076] S400: construct a gradient matrix variable based on the operation matrix variable.
[0077] Among them, each gradient parameter in the gradient matrix variable corresponds to a weight parameter in the weight matrix.
[0078] Specifically, since each weight parameter in the weight matrix corresponds to a row vector and a column vector, its gradient parameter may be the sum of the corresponding row vector and column vector.
[0079] Therefore, the gradient matrix variable can be constructed by taking the sum of the row vector and the column vector corresponding to a weight parameter in the weight matrix as a gradient parameter in the gradient matrix variable.
[0080] Since the operation matrix variable includes a row vector and a column vector, the functional relationship between the operation matrix variable and the gradient matrix variable can be obtained based on the matrix operation expression, and the gradient matrix variable can be obtained based on the functional relationship.
[0081] Specifically, see Figure 3 , Figure 3 yes Figure 1 A flow chart of an implementation method corresponding to S400 in FIG.
[0082] like Figure 3 As shown, the method of constructing a gradient matrix variable may include:
[0083] S401, construct an auxiliary matrix.
[0084] Specifically, the auxiliary matrix can satisfy: the sum of the products of the corresponding bits of any row or column of the operation matrix variable and any column or row of the auxiliary matrix is a gradient parameter; that is, the gradient matrix variable can be the product of the operation matrix variable and the auxiliary matrix.
[0085] To facilitate matrix multiplication operations, the operation matrix variable can be made into a (M+N)*1 matrix, the gradient matrix variable can be made into a MN*1 matrix, and an auxiliary matrix of MN*(M+N) can be constructed so that the product of the auxiliary matrix and the operation matrix is equal to the gradient matrix.
[0086] Exemplarily, the operation matrix variable may be:
[0087]
[0088] The gradient matrix variable can be:
[0089]
[0090] Among them, ΔW(m, n) can correspond to the gradient parameter of the weight parameter in the mth row and nth column in the weight matrix.
[0091] Correspondingly, the auxiliary matrix can be:
[0092]
[0093] S402: Based on the auxiliary matrix, a functional relationship between the operation matrix variable and the gradient matrix variable is obtained.
[0094] Based on the auxiliary matrix constructed in S401, the following functional relationship A*OP=ΔW can be obtained, where matrix A is the auxiliary matrix, matrix OP is the operation matrix variable, and matrix ΔW is the gradient matrix variable.
[0095] Specifically, taking the matrix in S501 as an example, the functional relationship may be:
[0096]
[0097] S500, taking the accurate gradient matrix as the target of gradient matrix variables, fitting and solving the operation matrix variables to obtain the optimal operation matrix.
[0098] The precise gradient matrix may be obtained based on the precise gradient information acquired in S100. Since the precise gradient information includes the precise gradient parameters of each weight parameter to be updated, the precise gradient parameters may be substituted into the gradient matrix variables to obtain the precise gradient matrix.
[0099] Substituting the precise gradient matrix as the target of the gradient matrix variable into the above functional relationship, the operation matrix OP with the minimum error that makes the above functional relationship valid can be obtained through the fitting algorithm, that is, the optimal operation matrix.
[0100] The following uses the least squares method as an example to introduce the calculation method of the optimal operation matrix.
[0101] First, based on the least squares method, we solve the above functional relationship A*OP=ΔW, and we can get the following equation: OP=(A T A) -1 *A T *ΔW.
[0102] Since the exact gradient matrix and the auxiliary matrix are known, based on the above equations, M*N sets of equations and (M+N) operating variables can be obtained, so that the optimal operating matrix can be calculated.
[0103] It should be noted that the above method only shows the method of fitting calculation by the least squares method. In other embodiments, other methods can also be used for fitting calculation, such as high-order polynomial fitting algorithm, Fourier series fitting algorithm, etc., which can also realize the fitting calculation of the above-mentioned functional relationship and achieve the effect of this embodiment.
[0104] In particular, in the least squares solution process, in order to optimize the solution, the perturbation matrix λI can also be introduced into the equation. Specifically, the equation for introducing the perturbation matrix λI can be OP=(A T A+λI) -1 *A T *ΔW.
[0105] S600: Update weight parameters of a weight matrix based on an optimal operation matrix.
[0106] Based on the optimal operation matrix obtained in S500, an update pulse of each row of weight parameters and an update pulse of each column of weight parameters in the weight matrix can be obtained, thereby realizing the update of the weight parameters.
[0107] Since the minimum unit of update operation is row and column, and the storage and computing integrated array shares row and column lines in a certain direction, the total number of operations can be reduced to M+1 or N+1, which greatly reduces power consumption and latency compared to the traditional one-by-one update method.
[0108] In some implementations, in order to perform performance evaluation on the above update method, the update method may further include:
[0109] S700: Obtain an approximate gradient matrix based on the optimal operation matrix.
[0110] The approximate gradient matrix includes the gradient parameters of each weight parameter in the weight matrix, that is, the update vector of each weight parameter.
[0111] Specifically, the method for calculating the approximate gradient matrix may be to substitute the optimal operation matrix into the above functional relationship A*OP=ΔW. Since the auxiliary matrix A and the optimal operation matrix OP are known, the approximate gradient matrix ΔW' may be calculated.
[0112] S800, calculating the mean square error between the approximate gradient matrix and the exact gradient matrix as a performance evaluation indicator.
[0113] It can be understood that the approximate gradient matrix is calculated based on the fitting algorithm, and the precise gradient matrix is obtained through the back propagation algorithm based on the error between the aligned label value and the neural network output value. By calculating the mean square error MSE (ΔW, ΔW') of the two, the performance of the algorithm of the present application can be evaluated.
[0114] Based on the update method of the above-mentioned implementation modes, when updating the M*N initial weight matrix of the neural network, the number of update operations is M+1 or N+1, and the types of update conditions are M+N. There is no need for external indexing of sparse information throughout the process. Compared with the traditional one-by-one update algorithm and row-column parallel update algorithm, the time and energy consumption in the sparse training process are greatly reduced.
[0115] In addition, the above update algorithm has an update accuracy and training effect comparable to the traditional precise update algorithm that updates unit by unit in the sparse neural network update process.
[0116] The following is a specific experiment to illustrate the effect of the technical solution of the present application. The vectorial approximate update algorithm (VAU) of the present application and the traditional cell-by-cell update algorithm are used to simulate and update the sparse neural network, and the correlation data of the mean square error (mean square error) and the sparsity (sparsity / %) of the approximate gradient matrix and the precise gradient matrix are recorded to obtain Figure 4 , Figure 4 This is the relationship between the mean square error and the sparsity rate of the VAU algorithm of this application.
[0117] At the same time, 13 sparse rate values ranging from 20% to nearly 90% are selected, and the sparse training accuracy (accuracy / %) of the VAU algorithm and the Cell-by-cell algorithm are compared through simulation. Figure 5 , Figure 5 This is a data chart comparing the sparse training accuracy of the VAU algorithm and the Cell-by-cell algorithm of this application.
[0118] like Figure 4 As shown in the figure, even at a lower sparsity, the mean square error between the approximate gradient matrix and the exact gradient matrix is much smaller than 1 state, that is, the error can be eliminated after quantization. Moreover, when the sparsity rate increases from 52.3% to 74.2%, the mean square error drops from 0.24 to 0.06, which is almost negligible. This is attributed to the large proportion of zero weights in sparse neural networks. Since the hardware adopts an index-free design, the updates of these zero weights are shielded, the number of equations in the least squares method is greatly reduced, and the approximation accuracy is greatly improved.
[0119] like Figure 5 As shown, the accuracy of the VAU algorithm of the present application is slightly lower than that of the traditional Cell-by-cell algorithm at a low sparsity rate (20%), while the accuracy of the VAU algorithm at other sparsity rates is comparable to that of the Cell-by-cell algorithm.
[0120] Therefore, the update algorithm of the present application has update accuracy and training effect comparable to the traditional precise update algorithm of updating unit by unit in the sparse neural network update process.
[0121] This application also provides a vector approximation updating device for a sparse neural network, see Figure 6 , Figure 6 It is a structural diagram of an implementation method of a vector approximation updating device for a sparse neural network of the present application.
[0122] like Figure 6 As shown, the device includes an acquisition module 21, a vectorization module 22, a first construction module 23, a second construction module 24, a fitting module 25, and an updating module 26.
[0123] Among them, the acquisition module 21 is used to obtain the precise gradient parameters of the weight parameters to be updated in the neural network, and obtain precise gradient information;
[0124] The vectorization module 22 is used to obtain the matrix structure of the weight matrix based on the precise gradient information, and construct the same update pulse for multiple weight parameters in the same row and multiple weight parameters in the same column in the matrix structure to obtain multiple row vectors and multiple column vectors;
[0125] The first construction module 23 is used to construct an operation matrix variable based on multiple row vectors and multiple column vectors;
[0126] The second construction module 24 is used to construct a gradient matrix variable based on the operation matrix variable, and each gradient parameter in the gradient matrix variable corresponds to a weight parameter in the weight matrix;
[0127] The fitting module 25 is used to fit and solve the operation matrix variables with the accurate gradient matrix as the target of the gradient matrix variable to obtain the optimal operation matrix;
[0128] The updating module 26 is used to update the weight parameters of the weight matrix based on the optimal operation matrix.
[0129] As above Figures 1 to 5 , a vector approximation updating method of a sparse neural network according to an embodiment of this specification is described. The details mentioned in the above description of the method embodiment are also applicable to the vector approximation updating device of a sparse neural network in an embodiment of this specification. The above vector approximation updating device of a sparse neural network can be implemented by hardware, or by software, or by a combination of hardware and software.
[0130] The present application also provides an electronic device, see Figure 7 , Figure 7 3 is a schematic diagram of the hardware structure of the electronic device of the present application. The electronic device 30 may include at least one processor 31, a memory 32 (e.g., a non-volatile memory), a memory 33, and a communication interface 34, and the at least one processor 31, the memory 32, the memory 33, and the communication interface 34 are connected together via a bus 35. At least one processor 31 executes at least one computer-readable instruction stored or encoded in the memory 32.
[0131] It should be understood that the computer executable instructions stored in the memory 32, when executed, cause at least one processor 31 to perform the above combined operations in various embodiments of the present specification. Figure 1-Figure 5 Describes the various operations and functions.
[0132] In the embodiments of the present specification, the electronic device 30 may include, but is not limited to, personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile electronic devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable electronic devices, consumer electronic devices, and the like.
[0133] According to one embodiment, a program product such as a machine-readable medium is provided. The machine-readable medium may have instructions (i.e., the above-mentioned elements implemented in the form of software), which, when executed by a machine, causes the machine to perform the above-mentioned combination of various embodiments of this specification. Figure 1-Figure 5 Specifically, a system or device equipped with a readable storage medium may be provided, on which a software program code implementing the functions of any of the above-mentioned embodiments is stored, and a computer or processor of the system or device reads and executes the instructions stored in the readable storage medium.
[0134] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of this specification.
[0135] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.
[0136] Those skilled in the art should understand that the various embodiments disclosed above can be modified and altered in various ways without departing from the essence of the invention. Therefore, the protection scope of this specification should be defined by the appended claims.
[0137] It should be noted that not all steps and units in the above-mentioned processes and system structure diagrams are necessary, and some steps or units can be ignored according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical client, or some units may be implemented by multiple physical clients, or some components in multiple independent devices may be implemented together.
[0138] In the above embodiments, the hardware unit or module can be realized by mechanical or electrical means. For example, a hardware unit, module or processor can include permanent dedicated circuit or logic (such as special processor, FPGA or ASIC) to complete the corresponding operation. The hardware unit or processor can also include programmable logic or circuit (such as general-purpose processor or other programmable processor), which can be temporarily set by software to complete the corresponding operation. Specific implementation (mechanical method or dedicated permanent circuit or temporary circuit) can be determined based on cost and time consideration.
[0139] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "exemplary" used throughout this specification means "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid making the concepts of the described embodiments difficult to understand, well-known structures and devices are shown in block diagram form.
[0140] The above description of the present disclosure is provided to enable any person of ordinary skill in the art to implement or use the present disclosure. Various modifications to the present disclosure will be apparent to those of ordinary skill in the art, and the general principles corresponding to the present disclosure may be applied to other variations without departing from the scope of protection of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the widest range of principles and novel features disclosed herein.
Claims
1. A vector approximation update method for sparse neural networks, It is characterized in that include: Obtain the precise gradient parameters of the weight parameters to be updated in the neural network and obtain precise gradient information; Based on the precise gradient information, a matrix structure of a weight matrix is obtained, and the same update pulse is constructed for multiple weight parameters in the same row and multiple weight parameters in the same column in the matrix structure to obtain multiple row vectors and multiple column vectors; Constructing an operation matrix variable based on the multiple row vectors and the multiple column vectors; Based on the operation matrix variable, construct a gradient matrix variable, each gradient parameter in the gradient matrix variable corresponds to a weight parameter in the weight matrix; Taking the accurate gradient matrix as the target of the gradient matrix variable, fitting and solving the operation matrix variable to obtain the optimal operation matrix; Based on the optimal operation matrix, weight parameters of the weight matrix are updated.
2. The vector approximation updating method according to claim 1, It is characterized in that The step of obtaining the precise gradient parameters of the weight parameters to be updated in the neural network is specifically as follows: The aligned label values are compared with the output values of the neural network, and based on the back propagation algorithm, the precise gradient parameters of the weight parameters to be updated in the neural network are obtained.
3. The vector approximation updating method according to claim 1, It is characterized in that The step of constructing a gradient matrix variable based on the operation matrix variable is specifically as follows: Constructing an auxiliary matrix so that the sum of the products of the corresponding bits of any row or column of the operation matrix variable and any column or row of the auxiliary matrix is a gradient parameter, and the gradient parameter is the sum of the row vector and the column vector corresponding to a weight parameter in the weight matrix; Based on the auxiliary matrix, a functional relationship A*OP=ΔW between the operation matrix variable and the gradient matrix variable is obtained, wherein matrix A is the auxiliary matrix, matrix OP is the operation matrix variable, and matrix ΔW is the gradient matrix variable.
4. The vector approximation updating method according to claim 3, It is characterized in that The weight matrix is an M*N matrix, the operation matrix variable is an (M+N)*1 matrix, the gradient matrix variable is an MN*1 matrix, and the auxiliary matrix is an MN*(M+N) matrix.
5. The vector approximation updating method according to claim 3, It is characterized in that The step of taking the accurate gradient matrix as the target of the gradient matrix variable, fitting and solving the operation matrix variable, and obtaining the optimal operation matrix is specifically as follows: Substituting the precise gradient information into the gradient matrix variable to obtain a precise gradient matrix; Solving the functional relationship based on the least squares method, we get the equation: OP = (A T A) -1 *A T *ΔW; Substituting the precise gradient matrix and the auxiliary matrix into the equation, the optimal operation matrix is calculated.
6. The vector approximation updating method according to claim 1, It is characterized in that Also includes: Based on the optimal operation matrix, an approximate gradient matrix is obtained; The mean square error between the approximate gradient matrix and the precise gradient matrix is calculated as a performance evaluation indicator.
7. A vector approximation updating device for a sparse neural network, It is characterized in that include: An acquisition module is used to obtain the precise gradient parameters of the weight parameters to be updated in the neural network and obtain precise gradient information; A vectorization module, used for obtaining a matrix structure of a weight matrix based on the precise gradient information, and constructing the same update pulse for multiple weight parameters in the same row and multiple weight parameters in the same column in the matrix structure to obtain multiple row vectors and multiple column vectors; A first construction module, configured to construct an operation matrix variable based on the plurality of row vectors and the plurality of column vectors; A second construction module is used to construct a gradient matrix variable based on the operation matrix variable, each gradient parameter in the gradient matrix variable corresponds to a weight parameter in the weight matrix; A fitting module is used to fit and solve the operation matrix variables with an accurate gradient matrix as the target of the gradient matrix variables to obtain an optimal operation matrix; An updating module is used to update the weight parameters of the weight matrix based on the optimal operation matrix.
8. An electronic device, include: at least one processor; as well as A memory storing instructions, which, when executed by the at least one processor, enable the at least one processor to perform the vector approximation updating method according to any one of claims 1 to 6.
9. A machine-readable storage medium storing executable instructions, wherein when the instructions are executed, the machine executes the vector approximation updating method according to any one of claims 1 to 6.
