Vector approximate update method for sparse neural network, and application thereof

Abstract

Disclosed in the present application are a vector approximate update method for a sparse neural network, and the application thereof. The method comprises: acquiring an exact gradient parameter of a weight parameter to be updated in a neural network, so as to obtain exact gradient information; constructing the same update pulse for a plurality of weight parameters in the same row and a plurality of weight parameters in the same column in a matrix structure; constructing an operation matrix variable; on the basis of the operation matrix variable, constructing a gradient matrix variable; with the objective of using an exact gradient matrix as the gradient matrix variable, performing fitting to solve the operation matrix variable, so as to obtain an optimal operation matrix; and on the basis of the optimal operation matrix, updating a weight parameter of a weight matrix. In the vector approximate update method in the present application, there is no need to perform the external indexing of sparse information throughout the process, thereby greatly reducing the time overhead and energy consumption overhead during a sparse training process; and during a sparse neural network update process, the update accuracy and training effect are comparable to those of conventional per-element updating exact update algorithms.

Description

Vector approximate updating method of sparse neural network and application

[0001] The present application claims priority to the Chinese patent application No. 2024109568184, filed on July 16, 2024, with the Chinese Patent Office, and entitled "Vector approximate updating method of sparse neural network and application", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of neural networks, in particular to a vector approximate updating method of sparse neural network and application. BACKGROUND

[0003] With the rapid development of artificial intelligence (AI), the scale of neural networks also grows exponentially. Model compression technology represented by sparse neural networks has gained unprecedented attention. The idea of sparse neural networks is to prune the weights with small values in dense neural networks, so that they do not participate in training and inference, thereby directly improving the computing speed and power consumption.

[0004] However, the ideal advantages of sparse neural networks have not been realized, partly because the traditional weight updating algorithm developed from dense networks is currently used for sparse neural network training, such as individual unit updating, row-column parallel updating, etc. The number of updates is large, the types of update conditions are many, and the external index of sparse information needs to be updated multiple times. The time cost and energy cost of updating are large, which not only fails to take advantage of sparsity, but also causes more additional overhead, masking the theoretical advantages of sparse neural networks. SUMMARY

[0005] The present application aims to provide a vector approximate updating method of sparse neural network and application, to solve the problem that the traditional weight updating algorithm developed from dense networks is used for sparse neural network training in the prior art, the time cost and energy cost of updating are large, the external index of sparse information needs to be updated multiple times, which not only fails to take advantage of sparsity, but also causes more additional overhead, masking the theoretical advantages of sparse neural networks.

[0006] To achieve the above-mentioned purpose, the first aspect of the present application provides a vector approximate updating method of sparse neural network, comprising:

[0007] obtaining the accurate gradient parameters of the weight parameters to be updated in the neural network to obtain the accurate gradient information;

[0008] Based on the accurate gradient information, a matrix structure of a weight matrix is obtained, and the same update pulse is constructed for a plurality of weight parameters in the same row and a plurality of weight parameters in the same column in the matrix structure, to obtain a plurality of row vectors and a plurality of column vectors;

[0009] Based on the plurality of row vectors and the plurality of column vectors, an operation matrix variable is constructed;

[0010] Based on the operation matrix variable, a gradient matrix variable is constructed, and each gradient parameter in the gradient matrix variable corresponds to a weight parameter in the weight matrix;

[0011] With an accurate gradient matrix as a target of the gradient matrix variable, the operation matrix variable is fitted and solved to obtain an optimal operation matrix;

[0012] Based on the optimal operation matrix, the weight parameters of the neural network are updated.

[0013] In one or more embodiments, the step of obtaining the accurate gradient parameter of the weight parameter to be updated in the neural network is specifically:

[0014] The registered label value and the output value of the neural network are compared, and based on the back propagation algorithm, the accurate gradient parameter of the weight parameter to be updated in the neural network is obtained.

[0015] In one or more embodiments, the step of constructing a gradient matrix variable based on the operation matrix variable is specifically:

[0016] An auxiliary matrix is constructed, so that the sum of the products of any row or column of the operation matrix variable and any column or row of the auxiliary matrix is a gradient parameter, and the gradient parameter is the sum of the row vector and the column vector corresponding to the weight parameter in the weight matrix;

[0017] Based on the auxiliary matrix, a functional relationship between the operation matrix variable and the gradient matrix variable is obtained, where matrix A is the auxiliary matrix, matrix OP is the operation matrix variable, and matrix ΔW is the gradient matrix variable.

[0018] In one or more embodiments, the weight matrix is matrix, the operation matrix variable is matrix, the gradient matrix variable is matrix, and the auxiliary matrix is matrix.

[0019] In one or more embodiments, the step of fitting and solving the operation matrix variable with an accurate gradient matrix as the target of the gradient matrix variable to obtain an optimal operation matrix is specifically:

[0020] substitute the accurate gradient information into the gradient matrix variable to obtain an accurate gradient matrix;

[0021] solving the function relationship based on the least square method to obtain an equation: .

[0022] substitute the accurate gradient matrix and the auxiliary matrix into the equation to obtain an optimal operation matrix.

[0023] In one or more embodiments, further comprising:

[0024] obtaining an approximate gradient matrix based on the optimal operation matrix;

[0025] calculating the mean square error of the approximate gradient matrix and the accurate gradient matrix as a performance evaluation index.

[0026] To achieve the above object, the second aspect of the present application provides a vector approximate update device of a sparse neural network, comprising:

[0027] an acquisition module configured to acquire an accurate gradient parameter of a weight parameter to be updated in a neural network to obtain accurate gradient information;

[0028] a vectorization module configured to acquire a matrix structure of a weight matrix based on the accurate gradient information, and construct the same update pulse for a plurality of weight parameters in the same row and a plurality of weight parameters in the same column in the matrix structure to obtain a plurality of row vectors and a plurality of column vectors;

[0029] a first construction module configured to construct an operation matrix variable based on the plurality of row vectors and the plurality of column vectors;

[0030] a second construction module configured to construct a gradient matrix variable based on the operation matrix variable, each gradient parameter in the gradient matrix variable corresponding to a weight parameter in the weight matrix;

[0031] a fitting module configured to fit and solve the operation matrix variable with the accurate gradient matrix as a target of the gradient matrix variable to obtain an optimal operation matrix;

[0032] an update module configured to update a weight parameter of the weight matrix based on the optimal operation matrix.

[0033] To achieve the above object, the third aspect of the present application provides an electronic device, comprising:

[0034] at least one processor; and

[0035] A memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the vector approximation update method according to any one of the preceding embodiments.

[0036] To achieve the above object, the fourth aspect of the present application provides a machine readable storage medium storing executable instructions, which, when executed, cause the machine to perform the vector approximation update method according to any one of the preceding embodiments.

[0037] Compared with the prior art, the present application has the following advantages:

[0038] The vector approximation update method of the present application is used for updating the weight matrix of a sparse neural network When updating, the number of update operations is M+1 or N+1, the number of update conditions is M+N, and no external indexing of sparse information is required throughout the process. Compared with the traditional individual update algorithm and row-column parallel update algorithm, the time and energy consumption of the sparse training process is greatly reduced.

[0039] The vector approximation update method of the present application has an update precision and training effect comparable to that of the traditional individual cell update precision algorithm in the sparse neural network update process. Even at a low sparsity, the mean square error between the approximate gradient matrix and the target matrix is much less than 1 state. When the sparsity increases from 52.3% to 74.2%, the mean square error decreases from 0.24 to 0.06, which can be ignored. Compared with the traditional individual cell update algorithm, the precision of the method of the present application is slightly lower than that of the traditional individual update algorithm at a low sparsity (20%), and the precision of the method of the present application is comparable to that of the individual update algorithm at other sparsities. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present application, and those skilled in the art can also obtain other drawings according to these drawings without creating any creative labor.

[0041] Fig. 1 is a flowchart of an embodiment of the vector approximation update method of the sparse neural network of the present application;

[0042] Fig. 2 is a schematic diagram of the vector approximation update operation of the 2T computing and storage unit of the present application;

[0043] Fig. 3 is a flowchart of an embodiment corresponding to S400 in Fig. 1;

[0044] Figure 4 is a diagram of the relationship between the mean square error and the sparsity of the VAU algorithm of the present application;

[0045] Figure 5 is a diagram of the comparison data of the sparsity training accuracy of the VAU algorithm of the present application and the Cell-by-cell algorithm;

[0046] Figure 6 is a structural schematic diagram of an embodiment of the vector approximate updating device of the sparse neural network of the present application;

[0047] Figure 7 is a hardware structural schematic diagram of the electronic device of the present application. DETAILED DESCRIPTION

[0048] In order for those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts should be within the scope of protection of the present application.

[0049] Inspired by the sparsity of the human brain, the idea of sparse neural network is to prune the weights with small values in dense neural network so that they do not participate in training and inference, thereby directly improving the computing speed and power consumption. For medium and large networks, the sparsity is often as high as 90% or more, which means a performance improvement of several orders of magnitude.

[0050] However, the ideal advantages of sparse neural networks have not been realized. One reason is that modern AI hardware relies on indexing of off-chip memory to complete pruning. Ideal fine-grained and unstructured pruning has too much indexing overhead, which directly masks the sparsity advantage, while structured and coarse-grained pruning reduces the indexing overhead but sacrifices accuracy. This is the accuracy-granularity dilemma faced by sparse neural networks, and there is currently no good solution.

[0051] On the other hand, for sparse neural networks, joint optimization for new hardware and algorithm innovation are still lacking. Traditional weight update algorithms developed from dense networks, such as cell-by-cell update and row-column parallel update, not only fail to take advantage of sparsity, but also cause more overhead, masking the theoretical advantages of sparse neural networks.

[0052] Specifically, in the cell-by-cell method, for a dense neural network of M rows and N columns, the number of updates required using cell-by-cell update is the number of update conditions For the same size of sparse network, the number of updates is (1-Sparsity) times, and the number of update conditions is (1-Sparsity) kinds, and in addition, the number of indexes for sparse information is times. This updating method is the most accurate, but also the most time-consuming and energy-consuming;

[0053] In the row-column parallel updating method, for an M-row N-column dense neural network, using column-wise updating, the number of updates is N times, and the number of update conditions required is kinds; for the same size of sparse network, although the number of updates is N times, the number of update conditions required is kinds, and in addition, the number of indexes for the whole column of sparse information is N times. Although this updating method is parallel and reduces the updating time, it still requires the peripheral to provide multiple update conditions and multiple external indexes for sparse information.

[0054] Therefore, the current sparse neural network updating method has large time cost and energy cost of updating, requires multiple external indexes for sparse information, and not only fails to take advantage of sparsity, but also causes more additional overhead, which masks the theoretical advantages of sparse neural networks.

[0055] In order to solve the above problems, the applicant has developed a new type of vectorial approximate update (Vectorial Approximate Update, abbreviated as VAU) algorithm for sparse neural networks. This updating algorithm can significantly reduce the number of weight updates, and does not require external indexing of sparse information throughout the process, greatly reducing the time and energy overheads in the sparse training process, and the algorithm has an update accuracy and training effect comparable to that of the single-unit-by-single-unit updating in the sparse network.

[0056] Specifically, please refer to FIG. 1, which is a flowchart of an embodiment of the vectorial approximate update method for sparse neural networks of the present application.

[0057] As shown in FIG. 1, the method comprises:

[0058] S100, obtaining the accurate gradient parameters of the weight parameters to be updated in the neural network, to obtain the accurate gradient information.

[0059] The accurate gradient parameters can be obtained by the backpropagation algorithm.

[0060] Specifically, the registered label value and the output value of the neural network can be compared to obtain the output error of the neural network.

[0061] The output error is back-propagated in the neural network, i.e., the error of each weight parameter to be updated in the neural network, i.e., the accurate gradient parameter, can be obtained.

[0062] S200, based on the accurate gradient information, obtaining the matrix structure of the weight matrix, and constructing the same update pulse for the plurality of weight parameters in the same row and the plurality of weight parameters in the same column in the matrix structure, to obtain a plurality of row vectors and a plurality of column vectors.

[0063] Since the accurate gradient information includes the accurate gradient parameters of all weight parameters to be updated, the number of weight parameters can be obtained based on the accurate gradient information, i.e., the matrix structure of the weight matrix.

[0064] For the weight matrix, the traditional update method is to construct a gradient for each weight parameter, and for the weight matrix, the traditional update method constructs gradients.

[0065] The method of the present application adopts the gradient vector scheme to replace the traditional gradient scheme, specifically, a consistent update pulse can be applied to all weight parameters in the mth row, which can be written as vector OP(m, ), thereby obtaining m row vectors, which can be expressed as:

[0066] .

[0067] Correspondingly, a consistent update pulse can be applied to all weight parameters in the nth column, which can be written as vector OP(, n), thereby obtaining n column vectors, which can be expressed as:

[0068] .

[0069] Based on the above steps, M+N gradient vectors are obtained, which are all operations required by the algorithm of the present application, and due to the sharing of the row and column lines in a certain direction of the storage and calculation integrated array, the total number of operations can be further reduced to M+1 or N+1.

[0070] For example, taking a 2T storage and calculation integrated unit as an example, referring to FIG. 2, which is a vectorization approximate update operation schematic diagram of the present application using a 2T storage and calculation integrated unit. As shown in FIG. 2, each unit in the storage and calculation integrated unit includes two ferroelectric transistors, one of which is used to control the switching of the other transistor, and the word lines of the units in the same row are shared. Since the word line is in the horizontal direction, M rows can be selected at the same time, and the M vectors of the row operation can be applied at the same time, and a total of N+1 operations are required.

[0071] ​It should be noted that the vectorization approximate updating method of the present application is not limited to the application scenario of the 2T storage and calculation integrated unit, and can also be applied to any device with storage and calculation integration capability, for example, it can also be applied to a memory device with a three-terminal structure. For example, each storage and calculation integrated unit can include a memory device with a three-terminal structure and a two-terminal structure RRAM, etc. The memory can achieve the effect of the present embodiment.

[0072] It can be understood that, compared with the traditional individual updating method, the method based on the above vectorization can reduce the power consumption and time delay by M times or N times.

[0073] S300, based on the plurality of row vectors and the plurality of column vectors, constructing an operation matrix variable.

[0074] For subsequent matrix processing, the plurality of row vectors and the plurality of column vectors obtained in S200 can be unified in a single matrix to obtain an operation matrix variable.

[0075] Taking the row vector and the column vector in S200 as an example, the operation matrix can be:

[0076] 。

[0077] S400, based on the operation matrix variable, constructing a gradient matrix variable.

[0078] Each gradient parameter in the gradient matrix variable corresponds to a weight parameter in the weight matrix.

[0079] Specifically, since each weight parameter in the weight matrix corresponds to a row vector and a column vector, the gradient parameter thereof can be the sum of the corresponding row vector and column vector.

[0080] Therefore, the sum of the row vector and the column vector corresponding to a weight parameter in the weight matrix can be taken as a gradient parameter in the gradient matrix variable to construct the gradient matrix variable.

[0081] Since the operation matrix variable includes row vectors and column vectors, the function relationship between the operation matrix variable and the gradient matrix variable can be obtained based on the matrix operation formula, and the gradient matrix variable can be obtained based on the function relationship.

[0082] Specifically, please refer to FIG. 3, which is a flowchart of an embodiment corresponding to S400 in FIG. 1.

[0083] As shown in FIG. 3, the method of constructing the gradient matrix variable can include:

[0084] S401, constructing an auxiliary matrix.

[0085] Specifically, the auxiliary matrix can satisfy: the sum of the product of any row or column of the operation matrix variable and the corresponding bit of any column or row of the auxiliary matrix is a gradient parameter; that is, the gradient matrix variable can be the product of the operation matrix variable and the auxiliary matrix.

[0086] To facilitate the matrix multiplication operation, the operation matrix variable can be the matrix, the gradient matrix variable is the matrix, and the auxiliary matrix of is constructed, so that the product of the auxiliary matrix and the operation matrix is equal to the gradient matrix.

[0087] For example, the operation matrix variable can be:

[0088] ;

[0089] The gradient matrix variable can be:

[0090] ;

[0091] Where ΔW(m, n) can correspond to the gradient parameter of the weight parameter in the mth row and the nth column of the weight matrix.

[0092] Correspondingly, the auxiliary matrix can be:

[0093] .

[0094] S402, based on the auxiliary matrix, obtaining the function relationship of the operation matrix variable and the gradient matrix variable.

[0095] Based on the auxiliary matrix constructed in S401, the following function relationship can be obtained , in the formula, the matrix A is the auxiliary matrix, the matrix OP is the operation matrix variable, and the matrix ΔW is the gradient matrix variable.

[0096] Specifically, taking the matrix in S501 as an example, the function relationship can be:

[0097] .

[0098] S500, taking the accurate gradient matrix as the gradient matrix variable, fitting and solving the operation matrix variable to obtain the optimal operation matrix.

[0099] Wherein, the accurate gradient matrix can be obtained based on the accurate gradient information obtained in S100, since the accurate gradient information includes the accurate gradient parameter of each weight parameter to be updated, the accurate gradient parameter can be substituted into the gradient matrix variable to obtain the accurate gradient matrix.

[0100] The precise gradient matrix is taken as a gradient matrix variable to substitute into the function relationship, and the operation matrix OP that minimizes the error of the function relationship can be obtained by a fitting algorithm, that is, the optimal operation matrix.

[0101] The following describes a calculation method of the optimal operation matrix by taking the least square method as an example.

[0102] First, the function relationship is solved based on the least square method , and the following equation can be obtained: .

[0103] Since the precise gradient matrix and the auxiliary matrix are known, the operation based on the above equation can obtain group equations and (M+N) operation variables, so that the optimal operation matrix can be calculated.

[0104] It should be noted that the above method only shows the fitting calculation method by the least square method, and other methods such as high-order polynomial fitting algorithm and Fourier series fitting algorithm can also be used for fitting calculation in other embodiments, which can also achieve the fitting calculation of the function relationship and achieve the effect of the embodiment.

[0105] In particular, in the least square method solving process, a perturbation matrix λI can also be introduced into the equation to optimize the solution. Specifically, the equation introducing the perturbation matrix λI can be .

[0106] S600, update the weight parameters of the weight matrix based on the optimal operation matrix.

[0107] Based on the optimal operation matrix obtained by S500, the update pulse of each row weight parameter and the update pulse of each column weight parameter in the weight matrix can be obtained, so that the update of the weight parameter is realized.

[0108] Since the minimum unit of the update operation is row and column, and the memory-compute integrated array shares the row and column lines in a certain direction, the total operation number can be reduced to M+1 or N+1, which greatly reduces the power consumption and time delay compared with the traditional update method.

[0109] In some embodiments, in order to evaluate the performance of the above update method, the update method can further include:

[0110] S700, obtain an approximate gradient matrix based on the optimal operation matrix.

[0111] The approximate gradient matrix includes the gradient parameters of each weight parameter in the weight matrix, that is, the update vector of each weight parameter.

[0112] Specifically, the method of calculating the approximate gradient matrix can be substituting the optimal operation matrix OP into the above function relationship Since the auxiliary matrix A is known and the optimal operation matrix OP is known, the approximate gradient matrix ΔW' can be calculated.

[0113] S800, calculate the mean square error of the approximate gradient matrix and the accurate gradient matrix as a performance evaluation index.

[0114] It can be understood that the approximate gradient matrix is calculated based on the fitting algorithm, and the accurate gradient matrix is obtained by back propagation algorithm based on the error between the registered label value and the neural network output value. By calculating the mean square error of the two , the performance of the algorithm of the present application can be evaluated.

[0115] Based on the update method of the above embodiments, when updating the initial weight matrix of the neural network , the number of update operations is M+1 or N+1, the number of update conditions is M+N, and the whole process does not need external indexing of sparse information. Compared with the traditional cell-by-cell update algorithm and row-column parallel update algorithm, the time and energy consumption of the sparse training process is greatly reduced.

[0116] In addition, the above update algorithm has the same update accuracy and training effect as the traditional cell-by-cell update algorithm in the sparse neural network update process.

[0117] The effect of the technical scheme of the present application will be described below in conjunction with specific experiments. The vectorial approximate update algorithm (Vectorial Approximate Update, abbreviated as VAU) of the present application and the traditional cell-by-cell update algorithm (Cell-by-cell) are used to simulate and update the sparse neural network, and the correlation data of the mean square error (mean square error) of the approximate gradient matrix and the accurate gradient matrix and the sparsity (sparsity / %) are recorded to obtain Figure 4. Figure 4 is a mean square error and sparsity relationship diagram of the VAU algorithm of the present application.

[0118] At the same time, 13 sparsity values from 20% to nearly 90% are selected, and the sparse training accuracy (accuracy / %) of the VAU algorithm and the Cell-by-cell algorithm is compared by simulation to obtain Figure 5. Figure 5 is a sparse training accuracy comparison data diagram of the VAU algorithm and the Cell-by-cell algorithm of the present application.

[0119] As shown in FIG. 4, even at a lower sparsity, the mean square error between the approximate gradient matrix and the accurate gradient matrix is much smaller than 1 state, that is, the error can be eliminated after quantization. Moreover, when the sparsity is increased from 52.3% to 74.2%, the mean square error is decreased from 0.24 to 0.06, which is almost negligible. This is due to the large proportion of zero weights in the sparse neural network. Since the hardware adopts an index-free design, the update of these zero weights is shielded, the number of equations in the least square method is greatly reduced, and the approximation accuracy is greatly improved.

[0120] As shown in FIG. 5, the VAU algorithm of the present application has a slightly lower accuracy than the traditional Cell-by-cell algorithm at a low sparsity (20%), but the accuracy of the VAU algorithm is comparable to that of the Cell-by-cell algorithm at other sparsities.

[0121] Therefore, the update algorithm of the present application has an update accuracy and training effect comparable to those of the traditional accurate update algorithm of the Cell-by-cell update.

[0122] The present application also provides a vector approximate update device for a sparse neural network. Please refer to FIG. 6, which is a structural schematic diagram of an embodiment of the vector approximate update device for a sparse neural network of the present application.

[0123] As shown in FIG. 6, the device comprises an acquisition module 21, a vectorization module 22, a first construction module 23, a second construction module 24, a fitting module 25, and an update module 26.

[0124] The acquisition module 21 is configured to acquire the accurate gradient parameters of the weight parameters to be updated in the neural network, to obtain accurate gradient information.

[0125] The vectorization module 22 is configured to acquire the matrix structure of the weight matrix based on the accurate gradient information, and to construct the same update pulse for the multiple weight parameters in the same row and the multiple weight parameters in the same column in the matrix structure, to obtain multiple row vectors and multiple column vectors.

[0126] The first construction module 23 is configured to construct an operation matrix variable based on the multiple row vectors and the multiple column vectors.

[0127] The second construction module 24 is configured to construct a gradient matrix variable based on the operation matrix variable, each gradient parameter in the gradient matrix variable corresponding to a weight parameter in the weight matrix.

[0128] The fitting module 25 is configured to fit the operation matrix variable with the accurate gradient matrix as the target of the gradient matrix variable, to obtain an optimal operation matrix.

[0129] The update module 26 is configured to update the weight parameters of the weight matrix based on the optimal operation matrix.

[0130] As described above with reference to FIGS. 1 to 5, the vector-approximate update method of the sparse neural network according to the embodiments of the present specification is described. The details mentioned in the above description of the method embodiments are also applicable to the vector-approximate update apparatus of the sparse neural network according to the embodiments of the present specification. The above vector-approximate update apparatus of the sparse neural network can be implemented in hardware, or implemented in software or a combination of hardware and software.

[0131] The present application also provides an electronic device, please refer to FIG. 7, FIG. 7 is a hardware structure schematic diagram of the electronic device of the present application. The electronic device 30 can include at least one processor 31, a memory 32 (such as a non-volatile memory), a memory 33 and a communication interface 34, and the at least one processor 31, the memory 32, the memory 33 and the communication interface 34 are connected together via a bus 35. The at least one processor 31 executes at least one computer readable instructions stored or encoded in the memory 32.

[0132] It should be understood that the computer executable instructions stored in the memory 32 when executed cause the at least one processor 31 to perform various operations and functions described above in conjunction with FIGS. 1-5 in various embodiments of the present specification.

[0133] In the embodiments of the present specification, the electronic device 30 can include but is not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile electronic devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable electronic devices, consumer electronic devices, and the like.

[0134] According to one embodiment, a program product such as a machine-readable medium is provided. The machine-readable medium can have instructions (i.e., the above-mentioned elements implemented in software) that, when executed by a machine, cause the machine to perform various operations and functions described above in conjunction with FIGS. 1-5 in various embodiments of the present specification. Specifically, a system or apparatus equipped with a readable storage medium on which a software program code implementing the functions of any of the above embodiments is stored, and a computer or processor of the system or apparatus can be provided to read and execute the instructions stored in the readable storage medium.

[0135] In this case, the program code read from the readable medium itself can implement the functions of any of the above embodiments, so the machine readable code and the readable storage medium storing the machine readable code constitute a part of the present specification.

[0136] Embodiments of the readable storage medium include floppy diskettes, hard disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-RWs), magnetic tapes, flash memories, and ROMs. The program code can also be downloaded from a server computer or cloud over a communication network.

[0137] Those skilled in the art will understand that various embodiments disclosed above can be modified and altered without departing from the spirit of the application. Therefore, the scope of protection of the present specification should be defined by the appended claims.

[0138] It should be noted that not all steps and units in the above processes and system structure diagrams are necessary, and some steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in each of the above embodiments can be a physical structure or a logical structure, i.e., some units can be implemented by the same physical client, or some units can be implemented by multiple physical clients, or some units can be implemented by some components in multiple independent devices.

[0139] In each of the above embodiments, a hardware unit or module can be implemented mechanically or electronically, for example, a hardware unit, module or processor can include permanent dedicated circuitry or logic (e.g., a dedicated processor, FPGA, or ASIC) to perform the corresponding operation. A hardware unit or processor can also include programmable logic or circuitry (e.g., a general processor or other programmable processor) that can be temporarily configured by software to perform the corresponding operation. The specific implementation (mechanical or dedicated permanent circuitry, or temporarily configured circuitry) can be determined based on cost and time considerations.

[0140] The specific implementation described above in conjunction with the accompanying drawings describes exemplary embodiments, but does not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "exemplary" used throughout this specification means "serving as an example, instance, or illustration" and does not mean "preferred" or "having advantages over other embodiments". The specific implementation includes specific details for the purpose of providing an understanding of the described technology. However, these technologies can be implemented without these specific details. In some instances, well-known structures and devices are shown in block diagrams to avoid obscuring the concepts of the described embodiments.

[0141] The foregoing description of the present disclosure has been presented for the purposes of conduction and enabling those of ordinary skill in the art to make and use the present disclosure. Modifications to embodiments of the present disclosure implementing the principles of the present disclosure can occur to those skilled in the art with the benefit of the present disclosure. Therefore, what has been described above is merely illustrative of the principles of the present disclosure, and the present disclosure is to be given a broad interpretation.

Claims

1. A method for vector approximate update of a sparse neural network, characterized in that, The method comprises the following steps: obtain the accurate gradient parameters of the weight parameters to be updated in the neural network to obtain accurate gradient information; based on the accurate gradient information, obtain the matrix structure of the weight matrix, and construct the same update pulse for the multiple weight parameters in the same row and the multiple weight parameters in the same column in the matrix structure to obtain multiple row vectors and multiple column vectors; based on the multiple row vectors and the multiple column vectors, construct an operation matrix variable; based on the operation matrix variable, construct a gradient matrix variable, each gradient parameter in the gradient matrix variable corresponds to a weight parameter in the weight matrix; take the accurate gradient matrix as the target of the gradient matrix variable, fit and solve the operation matrix variable to obtain an optimal operation matrix; update the weight parameters of the weight matrix based on the optimal operation matrix.

2. The vector approximation update method of claim 1, wherein, The step of obtaining the accurate gradient parameters of the weight parameters to be updated in the neural network is specifically: compare the registered label value and the output value of the neural network, and obtain the accurate gradient parameters of the weight parameters to be updated in the neural network based on the back propagation algorithm.

3. The vector approximation update method of claim 1, wherein, The step of constructing a gradient matrix variable based on the operation matrix variable is specifically: construct an auxiliary matrix, so that the sum of the product of any row or column of the operation matrix variable and any column or row of the auxiliary matrix is a gradient parameter, and the gradient parameter is the sum of the row vector and the column vector corresponding to a weight parameter in the weight matrix; based on the auxiliary matrix, obtain the functional relationship between the operation matrix variable and the gradient matrix variable, wherein matrix A is the auxiliary matrix, matrix OP is the operation matrix variable, and matrix ΔW is the gradient matrix variable.

4. The vector approximation update method of claim 3, wherein, The weight matrix is a matrix, the operation matrix variable is a matrix, the gradient matrix variable is a matrix, the auxiliary matrix is a matrix.

5. The vector approximation update method of claim 3, wherein, The step of taking the accurate gradient matrix as the target of the gradient matrix variable, fitting and solving the operation matrix variable to obtain an optimal operation matrix is specifically: substitute the accurate gradient information into the gradient matrix variable to obtain an accurate gradient matrix; Solving the functional relationship based on the least square method, the equation is obtained: ； substitute the accurate gradient matrix and the auxiliary matrix into the equation to calculate the optimal operation matrix.

6. The vector approximation update method of claim 1, wherein, It also comprises: based on the optimal operation matrix, obtain an approximate gradient matrix; calculate the mean square error of the approximate gradient matrix and the accurate gradient matrix as a performance evaluation index.

7. A vector approximation update device for a sparse neural network, characterized in that, The method comprises the following steps: an acquisition module is configured to obtain the accurate gradient parameters of the weight parameters to be updated in the neural network to obtain accurate gradient information; a vectorization module is configured to obtain the matrix structure of the weight matrix based on the accurate gradient information, and construct the same update pulse for the multiple weight parameters in the same row and the multiple weight parameters in the same column in the matrix structure to obtain multiple row vectors and multiple column vectors; a first construction module is configured to construct an operation matrix variable based on the multiple row vectors and the multiple column vectors; a second construction module is configured to construct a gradient matrix variable based on the operation matrix variable, and each gradient parameter in the gradient matrix variable corresponds to a weight parameter in the weight matrix; a fitting module is configured to take the accurate gradient matrix as the target of the gradient matrix variable, fit and solve the operation matrix variable to obtain an optimal operation matrix; an updating module configured to update a weight parameter of the weight matrix based on the optimal operation matrix.

8. An electronic device, comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the vector approximation updating method according to any one of claims 1 to 6.

9. A machine-readable storage medium having stored thereon executable instructions that, when executed by a machine, cause the machine to perform the vector approximation updating method according to any one of claims 1 to 6.

Images