A neural network calculation method and device

Abstract

The embodiment of the present application discloses a neural network calculation method and device, which relates to the field of communication technology, and solves the problem in the prior art that when the block size is large, the training is difficult to converge and the accuracy decreases significantly when the block size is large when the block circulant matrix compresses the neural network. The specific scheme is: obtain the input vector of the first network layer to be processed; obtain the perturbation vector of the first network layer to be processed according to the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed, wherein the dimension of the perturbation vector is equal to the dimension of the input vector of the first network layer to be processed; multiply the elements in the input vector of the first network layer to be processed by the elements in the corresponding positions in the perturbation vector to obtain the corrected input vector of the first network layer to be processed; obtain the output vector of the first network layer to be processed based on the corrected input vector and the calculation model of the neural network.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communication technology, and in particular to a neural network computing method and device. Background Art

[0002] Neural network is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. It has been widely used in many fields. At present, structured matrices have achieved good results in the compression and acceleration of neural networks. Among them, block circulant matrices are a type of structured matrix. Block circulant matrices are applied to convolutional layers, fully connected layers, and long short-term memory layers (Long Short-Term Memory, LSTM) to achieve high-multiple network parameter compression.

[0003] However, the existing block circulant matrix compression neural network method, when the block size is large, introduces correlation between output feature images due to the structuring of weight parameters, which causes information loss and leads to accuracy loss of application tasks based on neural networks, making the training convergence slow or even non-convergence, and reducing the performance of the model. Summary of the invention

[0004] The embodiments of the present application provide a neural network calculation method and device, which can solve the problem that when the block size is large, the training is difficult to converge and the accuracy drops significantly when the block size is large when the neural network is compressed by a block circulant matrix.

[0005] In order to achieve the above purpose, the embodiment of the present application adopts the following technical solution:

[0006] According to a first aspect of an embodiment of the present application, a neural network calculation method is provided, wherein the neural network includes multiple network layers, wherein the multiple network layers include a first network layer to be processed, and the method includes: obtaining an input vector of the first network layer to be processed; obtaining a perturbation vector of the first network layer to be processed according to a reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed, wherein the dimension of the perturbation vector is equal to the dimension of the input vector of the first network layer to be processed, and the dimension of the random vector of the first network layer to be processed is determined based on the weight parameters of the first network layer to be processed and the dimension of the input vector of the first network layer to be processed; multiplying the elements in the input vector of the first network layer to be processed with the elements at corresponding positions in the perturbation vector to obtain a corrected input vector of the first network layer to be processed; and obtaining an output vector of the first network layer to be processed based on the corrected input vector and the calculation model of the neural network. When the first network layer to be processed is the input layer of the neural network, the above-mentioned obtaining the input vector of the first network layer to be processed includes using the input vector of the neural network as the input vector of the first network layer to be processed; when the first network layer to be processed is not the input layer of the neural network, the above-mentioned obtaining the input vector of the first network layer to be processed includes using the output vector of the previous network layer of the first network layer to be processed as the input vector of the first network layer to be processed. Based on this solution, the correlation between the output feature images can be reduced, the training convergence speed can be accelerated, and the accuracy loss can be reduced.

[0007] In combination with the first aspect and the above possible implementation, in another possible implementation, the input vector of the first network layer to be processed is {x1, x2...xn}, the perturbation vector is {y1, y2...yn}, and the correction input vector of the first network layer to be processed is {x1×y1, x2×y2...xn×yn}, where n is a positive integer. Based on this solution, by adding a perturbation vector to the input vector of the first network layer to be processed, the correlation of the projection matrix of this layer can be effectively reduced.

[0008] In combination with the first aspect and the above possible implementation manner, in another possible implementation manner, the above neural network calculation model includes: j =C j ×X j +b j , where Y j is the output vector of the above neural network calculation model, X j is the input vector of the above neural network calculation model, C j is the weight parameter of the above neural network calculation model, b jis the preset bias value of the neural network calculation model, and the neural network calculation model is used for the neural network calculation of the jth layer among the multiple network layers, where j is an integer. The calculation model based on the correction input vector and the neural network is used to obtain the output vector of the first network layer to be processed, including: using the weight parameters and the correction input vector of the first network layer to be processed as the weight parameters and input vector of the neural network calculation model respectively to calculate, so as to obtain the output vector of the first network layer to be processed. Based on this solution, the output vector of the first network layer to be processed can be calculated by correcting the input vector and the weight parameters.

[0009] In combination with the first aspect and the above possible implementation, in another possible implementation, the weight parameter is a compressed matrix corresponding to a block structured matrix, wherein the block structured matrix is ​​uniquely determined by the compressed matrix, and before the above correction input vector is calculated by a neural network to obtain the output vector of the first network layer to be processed, the method further includes: decompressing the weight parameter into a block structured matrix corresponding to the compressed matrix. Based on this solution, for the first network layer to be processed whose weight parameter is a compressed matrix corresponding to a block structured matrix, the output vector can be calculated by decompressing the weight parameter to obtain the block structured matrix.

[0010] In combination with the first aspect and the above possible implementation, in another possible implementation, based on the weight parameters of the first network layer to be processed and the dimension of the input vector of the first network layer to be processed, the dimension of the random vector of the first network layer to be processed is determined, including: determining a target interval, the two interval endpoints of the target interval are the block size of the block structured matrix corresponding to the weight parameters of the first network layer to be processed and the dimension of the input vector of the first network layer to be processed; randomly determining the dimension of the random vector of the first network layer to be processed within the target interval. Based on this scheme, the dimension of the random vector of the first network layer to be processed can be obtained, and the dimension of the random vector is smaller than the dimension of the input vector.

[0011] In combination with the first aspect and the above possible implementations, in another possible implementation, the above multiple network layers include multiple first network layers to be processed, and before obtaining the perturbation vector of the above first network layer to be processed based on the reference random vector of the above neural network and the dimension of the random vector of the above first network layer to be processed, it also includes: determining the maximum value of the dimensions of the random vectors of all the above first network layers to be processed as the dimension of the reference random vector of the above neural network; generating random numbers that meet the dimension of the above reference random vector based on a preset random number generation model; and composing the above generated random numbers into a reference random vector. Based on this solution, the dimension of the reference random vector can be determined according to the dimension of the random vectors of all the above first network layers to be processed, and the dimension of the reference random vector is less than the dimension of the input vector.

[0012] In combination with the first aspect and the above-mentioned possible implementation methods, in another possible implementation method, according to the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed, the perturbation vector of the first network layer to be processed is obtained, including: intercepting a vector with the same dimension as the random vector of the first network layer to be processed in the reference random vector as the random vector of the first network layer to be processed; using a circular shift or vector element replication method for the random vector of the first network layer to be processed to generate a vector with the same dimension as the input vector of the first network layer to be processed as the perturbation vector of the first network layer to be processed. Based on this scheme, the perturbation vector of the first network layer to be processed can be obtained by intercepting and expanding the reference random vector and the dimension of the random vector of the first network layer to be processed, so only a few storage resources and computing resources need to be added to ensure that the accuracy of the neural network does not decrease and the convergence speed is effectively improved.

[0013] The second aspect of the embodiment of the present application provides a training method for a neural network model, which is used to obtain the weight parameters of each network layer in the neural network, and the method includes: step 1, initializing the weight parameters of each network layer; step 2, performing neural network calculation according to the neural network calculation model corresponding to each network layer to obtain the temporary output vector of the neural network, wherein the weight parameters of the neural network calculation model are the initialized weight parameters, and the neural network calculation according to the neural network calculation model corresponding to each network layer includes performing neural network calculation on one or more of the first to-be-processed network layers in each network layer according to the neural network calculation method of the first aspect or any implementation of the first aspect; step 3, updating the weight parameters of each network layer through the reverse transmission of the neural network; using the updated weight parameters of each network layer as the weight parameters of the neural network calculation model corresponding to each network layer, repeating steps 2 and 3 until the difference between the temporary output vector and the preset output vector of the neural network is less than the preset value; step 4, obtaining the weight parameters of each network layer. Based on this scheme, multiple network layers of the neural network can be trained to obtain the weight parameters of each network layer when the training converges.

[0014] According to a third aspect of an embodiment of the present application, a neural network computing device is provided, wherein the neural network includes a plurality of network layers, wherein the plurality of network layers include a first network layer to be processed, and the device includes: a first acquisition unit, used to acquire an input vector of the first network layer to be processed; a second acquisition unit, used to acquire a perturbation vector of the first network layer to be processed according to a reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed, wherein the dimension of the perturbation vector is equal to the dimension of the input vector of the first network layer to be processed, and the dimension of the random vector of the first network layer to be processed is determined according to a weight parameter of the first network layer to be processed and the dimension of the input vector of the first network layer to be processed; a first calculation unit, used to multiply an element in the input vector of the first network layer to be processed by an element at a corresponding position in the perturbation vector acquired by the second acquisition unit, so as to obtain a corrected input vector of the first network layer to be processed; and a second calculation unit, used to obtain an output vector of the first network layer to be processed based on the corrected input vector acquired by the first calculation unit and the computing model of the neural network. When the first network layer to be processed is the input layer of the neural network, the acquisition unit is specifically used to acquire the input vector of the neural network; when the first network layer to be processed is not the input layer of the neural network, the acquisition unit is specifically used to acquire the output vector of the previous network layer of the first network layer to be processed.

[0015] In combination with the third aspect and the above-mentioned possible implementation methods, in another possible implementation method, the input vector of the above-mentioned first network layer to be processed is {x1, x2...xn}, the above-mentioned disturbance vector is {y1, y2...yn}, and the correction input vector of the above-mentioned first network layer to be processed is {x1×y1, x2×y2...xn×yn}, where n is a positive integer.

[0016] In combination with the third aspect and the above possible implementation manner, in another possible implementation manner, the above neural network calculation model includes: j =C j ×X j +b j , where Y j is the output vector of the above neural network calculation model, X j is the input vector of the above neural network calculation model, C j is the weight parameter of the above neural network calculation model, b j is a preset bias value of the neural network calculation model, and the neural network calculation model is used for the neural network calculation of the jth layer among the multiple network layers, where j is an integer. The second calculation unit is specifically used to: calculate the weight parameters and the corrected input vector of the first network layer to be processed as the weight parameters and the input vector of the neural network calculation model, respectively, to obtain the output vector of the first network layer to be processed.

[0017] In combination with the third aspect and the above-mentioned possible implementation methods, in another possible implementation method, the above-mentioned weight parameters are compressed matrices corresponding to the block structured matrix, wherein the above-mentioned block structured matrix is ​​uniquely determined by the above-mentioned compressed matrix, and the above-mentioned device also includes a decompression unit, which is used to decompress the above-mentioned weight parameters into the block structured matrix corresponding to the above-mentioned compressed matrix.

[0018] In combination with the third aspect and the above-mentioned possible implementation methods, in another possible implementation method, the above-mentioned device also includes a first determination unit, which is used to determine a target interval, and the two interval endpoints of the target interval are respectively the block size of the block structured matrix corresponding to the weight parameters of the above-mentioned first network layer to be processed and the dimension of the input vector of the above-mentioned first network layer to be processed; the above-mentioned first determination unit is also used to randomly determine the dimension of the random vector of the above-mentioned first network layer to be processed within the above-mentioned target interval.

[0019] In combination with the third aspect and the above-mentioned possible implementation methods, in another possible implementation method, the above-mentioned multiple network layers include multiple first network layers to be processed, and the above-mentioned device also includes: a second determination unit and a generation unit, the second determination unit is also used to determine the maximum value of the dimensions of the random vectors of all the above-mentioned first network layers to be processed as the dimension of the reference random vector of the above-mentioned neural network; the above-mentioned generation unit is used to generate random numbers that meet the dimension of the above-mentioned reference random vector based on a preset random number generation model, and the generated random numbers constitute the above-mentioned reference random vector.

[0020] In combination with the third aspect and the above-mentioned possible implementation methods, in another possible implementation method, the above-mentioned second acquisition unit is specifically used to intercept a vector with the same dimension as the random vector of the above-mentioned first network layer to be processed in the above-mentioned reference random vector as the random vector of the above-mentioned first network layer to be processed; the above-mentioned second acquisition unit is specifically used to generate a vector with the same dimension as the input vector of the above-mentioned first network layer to be processed by using a cyclic shift or vector element replication method for the random vector of the above-mentioned first network layer to be processed as the disturbance vector of the above-mentioned first network layer to be processed.

[0021] According to a fourth aspect of an embodiment of the present application, a training device for a neural network model is provided, which is used to obtain weight parameters of each network layer in a neural network, and the device includes: an initialization unit, which is used to initialize the weight parameters of each network layer; a neural network calculation unit, which is used to perform neural network calculation according to the neural network calculation model corresponding to each network layer to obtain a temporary output vector of the neural network, wherein the weight parameters of the neural network calculation model are the initialized weight parameters; the neural network calculation unit is specifically used to perform neural network calculation on one or more of the first to-be-processed network layers in each network layer according to the neural network calculation method described in the first aspect or any implementation of the first aspect; a reverse transfer unit, which is used to update the weight parameters of each network layer through reverse transfer of the neural network; an acquisition unit, which is used to acquire the weight parameters of each network layer, and the weight parameters of each network layer are acquired when the difference between the temporary output vector of the neural network and the preset output vector of the neural network is less than a preset value.

[0022] The description of the effects of the third aspect and various implementation methods of the third aspect can refer to the description of the corresponding effects of the first aspect, and the description of the effects of the fourth aspect can refer to the description of the corresponding effects of the second aspect, which will not be repeated here.

[0023] In a fifth aspect of an embodiment of the present application, a server is provided, the structure of which includes a processor and a memory, the memory is used to couple with the processor to store program instructions and data necessary for the server, and the processor is used to execute the program instructions stored in the memory, so that the server executes the above method.

[0024] In a sixth aspect of an embodiment of the present application, a computer storage medium is provided, in which a computer program code is stored. When the computer program code runs on a processor, the processor executes the neural network calculation method described in the first aspect or any possible implementation of the first aspect, or executes the training method of the neural network model described in the second aspect.

[0025] In a seventh aspect of the embodiments of the present application, a computer program product is provided, which stores computer software instructions executed by the above-mentioned processor, and the computer software instructions include a program for executing the solutions described in the above-mentioned aspects.

[0026] In an eighth aspect of an embodiment of the present application, a device is provided, which exists in the form of a chip product. The structure of the device includes a processor and a memory. The memory is used to couple with the processor to store necessary program instructions and data for the device. The processor is used to execute the program instructions stored in the memory, so that the device performs the function of the neural network computing device or the neural network model training device in the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of a hardware architecture for neural network computing provided in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of another hardware architecture for neural network computing provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of another hardware architecture for neural network computing provided in an embodiment of the present application;

[0030] Figure 4 A flowchart of a neural network calculation method provided in an embodiment of the present application;

[0031] Figure 5 A flowchart of another neural network calculation method provided in an embodiment of the present application;

[0032] Figure 6 A flowchart of another neural network calculation method provided in an embodiment of the present application;

[0033] Figure 7 A flowchart of another neural network calculation method provided in an embodiment of the present application;

[0034] Figure 8 A flowchart of another neural network calculation method provided in an embodiment of the present application;

[0035] Fig. 9 A schematic diagram comparing the effects obtained by using the prior art and the neural network calculation method in the embodiment of the present application;

[0036] Fig.10 A flowchart of a training method for a neural network model provided in an embodiment of the present application;

[0037] Fig.11 A schematic diagram of the composition of a neural network computing device provided in an embodiment of the present application;

[0038] Fig.12 A schematic diagram of the composition of a neural network model training device provided in an embodiment of the present application;

[0039] Fig.13 A schematic diagram of the composition of another neural network computing device provided in an embodiment of the present application;

[0040] Fig.14 A schematic diagram of the composition of another neural network model training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to solve the problem in the prior art that when a block circulant matrix is ​​used to compress a neural network, training is difficult to converge and the accuracy drops significantly when the block size is large, an embodiment of the present application provides a neural network calculation method, which can reduce the correlation between output feature images, speed up training convergence, and reduce accuracy loss.

[0042] The neural network in the embodiment of the present application includes multiple network layers, and the multiple network layers include one or more first network layers to be processed. Wherein, when the first network layer to be processed is the input layer of the neural network, the input vector of the first network layer to be processed is the input vector of the neural network; when the first network layer to be processed is not the input layer of the neural network, the input vector of the first network layer to be processed is the output vector of the previous network layer of the first network layer to be processed. The calculation method for any network layer in one or more network layers to be processed can be applied to Figure 1 The hardware architecture is shown.

[0043] like Figure 1As shown, the hardware structure comprises: a first acquisition module 101, a second acquisition module 102, a first calculation module 103 and a second calculation module 104. The first acquisition module 101 is connected to the first calculation module 103 and the second calculation module 104 in sequence, and the second acquisition module 102 is connected to the first calculation module 103 and the second calculation module 104 in sequence.

[0044] The first acquisition module 101 is used to acquire the input vector of each first network layer to be processed. Exemplarily, for the first network layer, its input vector is the input vector of the neural network; for the network layers other than the first network layer, its input vector is the output vector of the previous network layer of the network layer.

[0045] The second acquisition module 102 is used to acquire a perturbation vector of the first network layer to be processed according to the input reference random vector and the dimension of the random vector of the first network layer to be processed, wherein the dimension of the perturbation vector of the first network layer to be processed may be equal to the dimension of the input vector of the first network layer to be processed. Exemplarily, the second acquisition module may include a cyclic shift module or a copy module.

[0046] The first calculation module 103 is used to add a disturbance based on the input vector of the first network layer to be processed obtained by the first acquisition module 101, and the disturbance is the disturbance vector obtained by the second acquisition module 102. Exemplarily, the corrected input vector of the first network layer to be processed can be obtained by multiplying the elements in the input vector of the first network layer to be processed with the elements in the corresponding positions in the disturbance vector.

[0047] The second calculation module 104 is used to calculate the output vector of the first network layer to be processed according to the correction input vector obtained by the first calculation module 103 and the calculation model of the neural network. Exemplarily, the calculation model of the neural network can be: j =C j ×X j +b j , where Y j is the output vector of the neural network calculation model, X j is the input vector of the neural network calculation model, C j is the weight parameter of the neural network calculation model, b j is a preset bias value of a neural network calculation model, and the neural network calculation model is used for neural network calculation of the jth layer in multiple network layers, where j is an integer. Exemplarily, the second calculation module can use a fast calculation method to calculate the output vector of the first layer to be processed. For example, when the block structured matrix is ​​a block circulant matrix, the second calculation module 104 can use a fast Fourier transform to calculate the output vector of the first layer to be processed.

[0048] Understandably, Figure 1 This is only an example. In actual applications, the hardware architecture of the neural network computing device may include: Figure 1 More or fewer components may be shown. Figure 1 The structure shown does not constitute any limitation to the hardware architecture provided in the embodiments of the present application.

[0049] Exemplarily, the hardware architecture in the embodiment of the present application consists of Figure 1 In addition to the modules shown, you can also include Figure 2 The decompression module 105 is shown, and the decompression module 105 is connected to the second calculation module 104 .

[0050] The decompression module 105 is used to decompress the weight parameters into a block-structured matrix corresponding to the compressed matrix, and transmit the block-structured matrix to the second calculation module 104 for calculating the output vector of the network layer. Exemplarily, when the block-structured matrix is ​​a block circulant matrix, the decompression module 105 can be a cyclic shift module, which decompresses the weight parameters into a block circulant matrix through the cyclicshift module, and then uses the second calculation module 104 to perform matrix multiplication to calculate the output vector of the first layer to be processed.

[0051] like Figure 3 As shown, an embodiment of the present application further provides a hardware structure, which includes a first determination module 301, a second determination module 302 and a generation module 303 connected in sequence.

[0052] The first determination module 301 is used to determine the dimension of the random vector of the first to-be-processed network layer according to the dimension of the input vector and the weight parameter.

[0053] The second determination module 302 is used to determine the dimension of the reference random vector according to the dimension of the random vector of one or more first to-be-processed network layers. Exemplarily, the dimension of the reference random vector may be the maximum dimension of the random vectors of all first to-be-processed network layers.

[0054] The generating module 303 is configured to generate a reference random vector having the same dimension as the reference random vector according to the dimension of the reference random vector.

[0055] Understandably, Figure 1-3 This is for illustrative purposes only. Figure 1-Figure 3 The structure shown does not constitute any limitation to the hardware architecture provided in the embodiments of the present application.

[0056] Combination Figure 1 , Figure 2 ,like Figure 4As shown, for any first to-be-processed network layer among the multiple network layers included in the neural network, the following method can be used: Figure 4 The calculation method shown in Calculate the output vector of the first network layer to be processed. Figure 4 As shown, the neural network calculation method provided in the embodiment of the present application may include steps S401-S404.

[0057] S401. Obtain an input vector of a first network layer to be processed.

[0058] Understandable, Figure 1 The first acquisition module 101 shown in FIG. 1 performs step S401 .

[0059] Exemplarily, the first network layer to be processed refers to a network layer whose weight parameter is a compression matrix corresponding to a block structured matrix among multiple network layers of a neural network. The block structured matrix can be uniquely determined by the compression matrix.

[0060] Exemplarily, the above-mentioned block structured matrix means that the matrix can be divided into multiple blocks, and each block is arranged according to a certain rule. The block structured matrix can include a block circulant matrix, a block Teplitz matrix, etc. The embodiment of the present application does not limit the specific type of the block structured matrix, and only the block circulant matrix is ​​used as an example for explanation.

[0061] For example, W is a block circulant matrix, in which the second column of each block is obtained by cyclically shifting the first column downward, and the third column is obtained by cyclically shifting the second column downward again. The weight parameter C of the network layer is the compression matrix corresponding to the block circulant matrix W. The compression matrix C is represented by a matrix composed of N w_bases, where N refers to the number of blocks of the block circulant matrix, and the w_base of each block is the minimum set of elements for each block of the structured matrix that uniquely determines the block (for example, w_base can be the first row or the first column of a block). The compression matrix of the block circulant matrix W in the following formula is C (weight parameter), and the block circulant matrix is ​​divided into 4 blocks. The block circulant matrix W can be uniquely determined by the compression matrix C.

[0062]

[0063] It can be understood that the neural network in the embodiment of the present application may include one or more first network layers to be processed, one of the one or more network layers to be processed can be the input vector of the neural network. When the first network layer to be processed is the first network layer of the neural network, the input vector of the first network layer to be processed is the input vector of the neural network; when the first network layer to be processed is any network layer other than the first network layer, the input vector of the first network layer to be processed is the output vector of the previous network layer of the first network layer to be processed.

[0064] Exemplarily, the neural network in the embodiment of the present application may be a deep neural network (DNN), a convolutional neural network (CNN), or other types of neural networks, and the first network layer to be processed may be a convolution layer, an LSTM layer, a full connection layer, etc. The embodiment of the present application does not limit the specific type and structure of the neural network.

[0065] S402: Obtain a disturbance vector of the first network layer to be processed according to the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed.

[0066] Understandable, Figure 1 The second acquisition module 102 shown in FIG. 1 performs step S402 .

[0067] The dimension of the perturbation vector is equal to the dimension of the input vector of the first network layer to be processed.

[0068] Exemplarily, the step S402 of obtaining the disturbance vector of the first network layer to be processed may include steps S4021-S4022.

[0069] S4021. Extract a vector having the same dimension as the random vector of the first network layer to be processed from the reference random vector as the random vector of the first network layer to be processed.

[0070] Exemplarily, the reference random vector may be the maximum value among the dimensions of all random vectors of the first network layer to be processed, and thus a vector having the same dimension as the random vector of the first network layer to be processed may be intercepted from the reference random vector as the random vector of the first network layer to be processed.

[0071] Exemplarily, the above-mentioned intercepting a vector with the same dimension as the random vector of the first network layer to be processed in the reference random vector may include: starting from a certain position (for example, the first position) of the reference random vector, sequentially intercepting a vector with the same dimension as the random vector of the first network layer to be processed as the random vector of the first network layer to be processed, or according to a certain interception rule, for example, intercepting a vector with the same dimension as the random vector of the first network layer to be processed from the odd or even bits in the reference random vector as the random vector of the first network layer to be processed. The embodiment of the present application does not limit the specific interception position and method of intercepting the random vector of the first network layer to be processed from the reference random vector, and is only an exemplary description here.

[0072] S4022: Generate a vector having the same dimension as the input vector of the first network layer to be processed by cyclic shift or vector element replication for the random vector of the first network layer to be processed as a disturbance vector of the first network layer to be processed.

[0073] Exemplarily, the dimension of the random vector of the first network layer to be processed is smaller than the dimension of the input vector of the first network layer to be processed, and the dimension of the perturbation vector of the first network layer to be processed is equal to the dimension of the input vector of the first network layer to be processed. Therefore, the random vector of the first network layer to be processed can be expanded by cyclic shift or vector element replication to obtain a vector with the same dimension as the input vector of the first network layer to be processed as the perturbation vector of the first network layer to be processed.

[0074] For example, if the random vector of the first network layer to be processed is A i =[1, -1, 1, 1, -1], the dimension of the input vector of the first network layer to be processed is 15, and the perturbation vector of the first network layer to be processed obtained by cyclic shift can be A_L i =[1,-1,1,1,-1,-1,1,1,-1,1,1,-1,1,-1,1]; the disturbance vector A_L of the first network layer to be processed is obtained by vector element replication i It can be A_L i =[1, -1, 1, 1, -1, 1, -1, 1, -1, 1, -1, 1, -1, 1, -1]. The embodiment of the present application does not limit the specific method for obtaining the disturbance vector of the first network layer to be processed according to the random vector of the first network layer to be processed.

[0075] It can be understood that when the perturbation vector of the first network layer to be processed is obtained by cyclic shift, Figure 1 The second acquisition module in the embodiment includes a cyclic shift module. When the disturbance vector of the first network layer to be processed is obtained by copying the vector elements, Figure 1The second acquisition module includes a copy module.

[0076] It should be noted that in the embodiment of the present application, only the reference random vector of the neural network and the dimension of the random vectors of all the first network layers to be processed need to be used to generate a perturbation vector with the same dimension as the input vector. Since the dimension of the random vector of the first network layer to be processed is smaller than the dimension of the input vector, only very small storage resources and computing resources need to be added to ensure that the accuracy of the neural network does not decrease and the convergence speed is effectively improved.

[0077] S403: multiply the elements in the input vector of the first network layer to be processed by the elements at corresponding positions in the disturbance vector to obtain a corrected input vector of the first network layer to be processed.

[0078] Understandable, Figure 1 The first calculation module 103 shown in FIG. 1 performs step S403 .

[0079] Exemplarily, if the input vector of the first network layer to be processed is {x1, x2...xn}, the perturbation vector of the first network layer to be processed is {y1, y2...yn}, the correction input vector of the first network layer to be processed is {x1×y1, x2×y2...xn×yn}, where n is a positive integer.

[0080] It should be noted that, in the embodiment of the present application, by adding a perturbation vector to the input vector of the first network layer to be processed, the correlation of the projection matrix of the layer can be effectively reduced.

[0081] S404: Based on the corrected input vector and the computing model of the neural network, an output vector of the first network layer to be processed is obtained.

[0082] Understandable, Figure 1 The second calculation module 104 shown in FIG. 1 performs step S404 .

[0083] Exemplarily, the above neural network calculation model includes: j =C j ×X j +b j , where Y j is the output vector of the neural network calculation model, X j is the input vector of the neural network calculation model, C j The weight parameters of the neural network calculation model, b j is the preset bias value of the neural network calculation model, "×" is a matrix multiplication operation, and the neural network calculation model is used for the neural network calculation of the jth layer in multiple network layers, and j is an integer.

[0084] It can be understood that the neural network calculation method provided in the embodiment of the present application can be applied to the calculation model to be converted into Y j =C j ×X j +b j For example, for the fully connected layer, its calculation model is the same as the above calculation model; for the LSTM layer, it can be composed of multiple above calculation models; for the convolutional layer, its original calculation model is Y j =C j *X j +b j , where "*" is a convolution operation, but according to the prior art, the convolution operation can be converted into a matrix multiplication operation, that is, the convolution layer can also be converted into the above calculation model. Therefore, the neural network calculation method in the embodiment of the present application can be applied to network layers including fully connected layers, convolution layers, LSTM, etc.

[0085] It should be noted that when compressing the convolution layer, after converting the convolution operation into matrix multiplication, the dimension of the input vector of the first network layer to be processed is the width of the input matrix after the im2col conversion according to the convolution kernel size. The im2col refers to expanding the pixel values ​​of a small window on the input image to be processed by the convolution kernel each time to a row (column) of the new matrix. The number of columns (rows) of the new matrix is ​​the number of convolution operations (the number of times the convolution kernel slides) for a pair of input images.

[0086] Exemplarily, the above-mentioned calculation model based on the corrected input vector and the neural network to obtain the output vector of the first network layer to be processed may include: calculating the weight parameters and the corrected input vector of the first network layer to be processed as the weight parameters and the input vector of the neural network calculation model respectively to obtain the output vector of the first network layer to be processed.

[0087] Exemplarily, the corrected input vector X′ obtained by adding the perturbation vector j and weight parameter C j As the weight parameter and input vector of the neural network calculation model, the output vector of the neural network calculation model is Y j =C j ×X′ j +b j , X′ j is the correction input vector of the neural network calculation model, Y j is the output vector of the first network layer to be processed.

[0088] The neural network calculation method provided in the embodiment of the present application obtains the input vector of the first network layer to be processed; obtains the perturbation vector of the first network layer to be processed according to the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed, wherein the dimension of the perturbation vector is equal to the dimension of the input vector of the first network layer to be processed; multiplies the elements in the input vector of the first network layer to be processed by the elements in the corresponding positions in the perturbation vector to obtain the corrected input vector of the first network layer to be processed; based on the corrected input vector and the calculation model of the neural network, the output vector of the first network layer to be processed is obtained. The embodiment of the present application adds the perturbation vector to the input vector of the first network layer to be processed, which can reduce the correlation between the output feature images, speed up the training convergence speed, and reduce the precision loss. Moreover, the embodiment of the present application can generate a perturbation vector with the same dimension as the input vector of the first network layer to be processed through the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed. Since the dimension of the random vector of the first network layer to be processed is smaller than the dimension of the input vector, only a very small amount of storage resources and computing resources need to be added to ensure that the accuracy and convergence speed of the neural network are effectively improved.

[0089] This application also provides an embodiment, such as Figure 5 As shown, before the above step S404, step S405 may also be included.

[0090] S405 , decompressing the weight parameters of the first network layer to be processed into a block-structured matrix corresponding to the compressed matrix.

[0091] Understandable, Figure 2 The decompression module 105 shown in FIG. 1 executes step S405 .

[0092] Exemplarily, since the weight parameter of the first network to be processed is a compressed matrix corresponding to the block structured matrix, before calculating the output vector of the first network layer to be processed, the weight parameter C j Decompress and obtain the block structured matrix W corresponding to the weight parameter (compression matrix) j , and then use the block structured matrix and the input vector to perform matrix multiplication to calculate the output vector.

[0093] Exemplarily, the output vector of the first network layer to be processed is specifically: Y m =W m ×X′ m +b m , where Y m is the output vector of the first network layer to be processed, X′ m is the correction input vector of the first network layer to be processed, W mis the block structured matrix corresponding to the weight parameters of the first network layer to be processed, b m is the preset bias value of the first network layer to be processed.

[0094] For example, if Figure 2 The decompression module 105 in the weight parameter decompresses the weight parameter into a block structured matrix (such as Figure 5 The calculation method shown in the figure can be obtained by Figure 2 The second calculation module 104 in performs matrix multiplication operation to obtain the output vector of the first network layer to be processed. Figure 4 The calculation method shown in Figure 1 The second calculation module 104 in the method uses fast Fourier transform to calculate the output vector of the first network layer to be processed.

[0095] The neural network calculation method provided in the embodiment of the present application obtains the input vector of the first network layer to be processed; obtains the perturbation vector of the first network layer to be processed according to the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed, wherein the dimension of the perturbation vector is equal to the dimension of the input vector of the first network layer to be processed; multiplies the elements in the input vector of the first network layer to be processed with the elements in the corresponding positions in the perturbation vector to obtain the corrected input vector of the first network layer to be processed; decompresses the weight parameters of the first network layer to be processed into a block-structured matrix corresponding to the compression matrix; and obtains the output vector of the first network layer to be processed based on the corrected input vector and the calculation model of the neural network. The embodiment of the present application can reduce the correlation between the output feature images, accelerate the training convergence speed, and reduce the precision loss by adding the perturbation vector to the input vector of the first network layer to be processed. Moreover, the embodiment of the present application can generate a perturbation vector with the same dimension as the input vector of the first network layer to be processed through the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed. Since the dimension of the random vector of the first network layer to be processed is smaller than the dimension of the input vector, only very little storage resources and computing resources need to be added to ensure that the accuracy and convergence speed of the neural network are effectively improved.

[0096] This application provides another embodiment, such as Figure 6 As shown, the above method may further include steps S601-S603.

[0097] S601. Determine the dimension of a random vector of a first network layer to be processed according to a weight parameter of the first network layer to be processed and the dimension of an input vector of the first network layer to be processed.

[0098] Understandable, Figure 3 The first determining module 301 shown in FIG. 1 performs step S601 .

[0099] Exemplarily, the above-mentioned determination of the dimension of the random vector of the first network layer to be processed based on the weight parameters of the first network layer to be processed and the dimension of the input vector of the first network layer to be processed may include: determining a target interval, the two interval endpoints of the target interval are the block size of the block structured matrix corresponding to the weight parameters of the first network layer to be processed and the dimension of the input vector of the first network layer to be processed; and randomly determining the dimension of the random vector of the first network layer to be processed within the target interval. For example, if the block size of the block circulant matrix of the first network layer to be processed in the current neural network is 32, and the dimension of the input vector of the neural network is 800, the dimension of the random vector of the first network layer to be processed can be a randomly selected integer in the interval (32, 800), for example, 40.

[0100] It can be understood that the neural network in the embodiment of the present application may include one or more first network layers to be processed. When the neural network includes multiple first network layers to be processed, the dimensions of the random vectors of the multiple first network layers to be processed may be the same or different, and the embodiment of the present application does not limit this.

[0101] S602: Determine the maximum value among the dimensions of all random vectors of the first to-be-processed network layers as the dimension of the reference random vector of the neural network.

[0102] Understandable, Figure 3 The second determination module 302 shown in FIG. 1 executes step S602 .

[0103] Exemplarily, when the multiple network layers of the neural network include multiple first network layers to be processed, the maximum value among the dimensions of the random vectors of all the first network layers to be processed is determined as the dimension of the reference random vector of the neural network. For example, when the neural network includes 5 first network layers to be processed, the dimensions of the random vectors of these 5 first network layers to be processed are 40, 35, 50, 46, and 55, respectively. Then the dimension of the reference random vector of the neural network is the maximum value of the dimensions of the random vectors of these 5 first network layers to be processed, that is, 55.

[0104] S603: Generate a random number that satisfies the dimension of the reference random vector based on a preset random number generation model.

[0105] Understandable, Figure 3 The generation module 303 shown in FIG. 1 executes step S603 .

[0106] Exemplarily, a set of random numbers that meet the dimension of the reference random vector can be generated according to a preset random number generation model and the dimension of the above-mentioned reference random vector. The random numbers that meet the dimension of the reference random vector are the reference random vector. For example, the random number generation model can be a symbol vector that obeys a binomial distribution. If the dimension of the reference random vector is 55, a symbol vector [1, -1, 1, 1, -1...1, -1] with a dimension of 55 can be generated. The reference random vector can be recorded as A, and A can be A = [1, -1, 1, 1, -1...1, -1]. The embodiments of the present application do not limit the specific form of the random number generation model. Any random number generation model is within the protection scope of the embodiments of the present application. This is only an exemplary description.

[0107] It should be noted that the process of determining the basic random vector in steps S601-S603 in the embodiment of the present application can be performed before the training of the neural network model described below. When performing forward calculation on the neural network, the output vector of the neural network can be calculated using the neural network calculation method of steps S401-S406 based on the weight parameters of each network layer obtained from the neural network model training, the reference random vector generated before model training, and the dimension of the random vector of the first network layer to be processed.

[0108] The neural network calculation method provided in the embodiment of the present application obtains the input vector of the first network layer to be processed; obtains the perturbation vector of the first network layer to be processed according to the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed, wherein the dimension of the perturbation vector is equal to the dimension of the input vector of the first network layer to be processed; multiplies the elements in the input vector of the first network layer to be processed by the elements in the corresponding positions in the perturbation vector to obtain the corrected input vector of the first network layer to be processed; based on the corrected input vector and the calculation model of the neural network, the output vector of the first network layer to be processed is obtained. The embodiment of the present application adds the perturbation vector to the input vector of the first network layer to be processed, which can reduce the correlation between the output feature images, speed up the training convergence speed, and reduce the precision loss. Moreover, the embodiment of the present application can generate a perturbation vector with the same dimension as the input vector of the first network layer to be processed through the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed. Since the dimension of the random vector of the first network layer to be processed is smaller than the dimension of the input vector, only a very small amount of storage resources and computing resources need to be added to ensure that the accuracy and convergence speed of the neural network are effectively improved.

[0109] The present application also provides another embodiment, when the neural network includes multiple first network layers to be processed, different first network layers to be processed in the multiple first network layers to be processed can use the same calculation method (such as Figure 4 or Figure 5When a plurality of different first network layers to be processed use different calculation methods, some of the first network layers to be processed may use Figure 5 and Figure 6 The calculation method shown in the figure, the other part of the first network layer to be processed can be used Figure 7 The calculation method shown in Figure 7 As shown, the method includes steps S701-S703.

[0110] S701. Obtain an input vector of a first network layer to be processed.

[0111] Understandable, Figure 1 or Figure 2 The first acquisition module 101 shown in FIG. 1 performs step S701 .

[0112] S702: Decompress the weight parameters of the first network layer to be processed into a block-structured matrix corresponding to the compressed matrix.

[0113] Understandable, Figure 2 The decompression module 105 shown in FIG. 1 executes step S702 .

[0114] Exemplarily, since the weight parameter of the first network to be processed is a compressed matrix corresponding to the block structured matrix, before calculating the output vector of the first network layer to be processed, the weight parameter C n Decompress and obtain the block structured matrix W corresponding to the weight parameter (compression matrix) n .

[0115] S703. Obtain an output vector of the first network layer to be processed according to the input vector of the first network layer to be processed and the calculation model of the neural network.

[0116] Understandable, Figure 1 or Figure 2 The second calculation module 104 shown in FIG. 1 performs step S703 .

[0117] Exemplarily, the output vector of the first network layer to be processed is Y n =W n ×X n +b n , where Y n is the output vector of the first network layer to be processed, X n is the input vector of the first network layer to be processed, W n is the block structured matrix corresponding to the weight parameters of the first network layer to be processed, b n is the preset bias value of the first network layer to be processed.

[0118] It should be noted that which of the one or more first processing network layers included in the neural network adopts Figure 5 The calculation method shown calculates the output of the network layer, which first network layer to be processed adopts Figure 7 The calculation method shown in the figure calculates the output of the network layer, which is not limited in the present embodiment and can be determined according to the actual network structure. However, when the neural network calculation method of the present embodiment is used, at least one first network layer to be processed should be used. Figure 5 The calculation method shown calculates the output vector of this network layer.

[0119] The neural network calculation method provided in the embodiment of the present application obtains the input vector of the first network layer to be processed; obtains the perturbation vector of the first network layer to be processed according to the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed, wherein the dimension of the perturbation vector is equal to the dimension of the input vector of the first network layer to be processed; multiplies the elements in the input vector of the first network layer to be processed by the elements in the corresponding positions in the perturbation vector to obtain the corrected input vector of the first network layer to be processed; based on the corrected input vector and the calculation model of the neural network, the output vector of the first network layer to be processed is obtained. The embodiment of the present application adds the perturbation vector to the input vector of the first network layer to be processed, which can reduce the correlation between the output feature images, speed up the training convergence speed, and reduce the precision loss. Moreover, the embodiment of the present application can generate a perturbation vector with the same dimension as the input vector of the first network layer to be processed through the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed. Since the dimension of the random vector of the first network layer to be processed is smaller than the dimension of the input vector, only a very small amount of storage resources and computing resources need to be added to ensure that the accuracy and convergence speed of the neural network are effectively improved.

[0120] The present application also provides another embodiment, if the neural network includes multiple network layers in addition to the first network layer to be processed, and also includes one or more second network layers to be processed, the output vector of any second network layer to be processed can be calculated by the calculation method in the embodiment of the present application. Figure 8 As shown, the calculation method may include steps S801-S802.

[0121] S801. Obtain an input vector of a second network layer to be processed.

[0122] Understandable, Figure 1 or Figure 2 The first acquisition module 101 shown in FIG. 8 performs step S801 .

[0123] The second network layer to be processed refers to a network layer whose weight parameter is an unstructured matrix among multiple network layers of the neural network. The unstructured matrix means that the matrix is ​​not arranged according to a certain rule and cannot be represented by the minimum element set that uniquely determines the structured matrix. The matrix A in the following formula is an unstructured matrix, which cannot be compressed and there is no compressed matrix.

[0124]

[0125] Exemplarily, when the second network layer to be processed is the first network layer of the neural network, the input vector of the second network layer to be processed is the input vector of the neural network; when the second network layer to be processed is any network layer except the first network layer, the input vector of the second network layer to be processed is the output vector of the previous network layer of the second network layer to be processed.

[0126] S802. Obtain an output vector of the second network to be processed according to the input vector of the second network layer to be processed and the calculation model of the neural network.

[0127] Understandable, Figure 1 or Figure 2 The second calculation module 104 shown in FIG. 8 performs step S802 .

[0128] Exemplarily, the output vector of the second network to be processed is Y i =C i ×X i +b i , where Y i is the output vector of the second network layer to be processed, X i is the input vector of the second network layer to be processed, C i is the weight parameter of the second network layer to be processed, b i is the preset bias value of the second network layer to be processed. Since the weight parameter of the second network layer to be processed is an unstructured matrix, C i =W i Therefore, the output vector of the second network layer to be processed can be directly calculated according to the input vector and weight parameters of the second network layer to be processed and the preset bias value, without decompressing the weight parameters.

[0129] like Fig. 9 As shown in the figure, it is the effect diagram of the experiment on the convolutional neural network consisting of 2 convolution layers and 1 fully connected layer on the MNIST database, where the second convolution layer is the first network layer to be processed, and the first convolution layer and the third fully connected layer are the second network layer to be processed. Fig. 9The horizontal axis in the figure represents the number of iterations, the vertical axis represents the accuracy, and the × and the straight line represent the effect diagram obtained by using the method of not adding a symbol vector in the prior art when the block size of the block circulant matrix is ​​32; the straight line represents the perturbation vector obtained by using the cyclic shift method when the block size of the block circulant matrix is ​​32. For the first network layer to be processed, Figure 5 The neural network calculation method shown in the figure adopts Figure 8 The effect diagram obtained by the neural network calculation method shown in the figure; the triangle with a straight line in the middle indicates that when the block size of the block circulant matrix is ​​40, the perturbation vector is obtained by vector element replication (content replication). For the first network layer to be processed, Figure 5 The neural network calculation method shown in the figure adopts Figure 8 The effect diagram obtained by the neural network calculation method shown in the figure; the dotted line indicates that when the block size of the block circulant matrix is ​​40, the perturbation vector is obtained by cyclic shift. For the first network layer to be processed, Figure 5 The neural network calculation method shown in the figure adopts Figure 8 The effect diagram obtained by the neural network calculation method shown.

[0130] Combination Fig. 9 As shown, when the block size is 32, due to the large block size, the network cannot converge when the calculation method without adding symbol vectors in the prior art is used, so 32 times compression cannot be achieved. However, the perturbation symbol vector of the first network layer to be processed is obtained by cyclic shift in the embodiment of the present application. According to the neural network calculation method of the above embodiment, the network can converge quickly, and the beneficial effect compared with the prior art is very obvious. Moreover, when the block size is 40, the perturbation vector of the first network layer to be processed is generated by content replication (vector element replication) and content cyclic shift in the embodiment of the present application, and the above network layer is used. Figure 5 and Figure 8 When the calculation method shown is used, the network can converge, and when cyclic shift is used, the network converges faster and with higher accuracy. Therefore, when the neural network calculation method in the embodiment of the present application is used to perform forward reasoning on the network, the problem that when the block size is large when the block circulant matrix is ​​used to compress the neural network, the training is difficult to converge and the accuracy drops significantly can be solved.

[0131] The neural network calculation method provided in the embodiment of the present application obtains the input vector of the first network layer to be processed; obtains the perturbation vector of the first network layer to be processed according to the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed, wherein the dimension of the perturbation vector is equal to the dimension of the input vector of the first network layer to be processed; multiplies the elements in the input vector of the first network layer to be processed by the elements in the corresponding positions in the perturbation vector to obtain the corrected input vector of the first network layer to be processed; based on the corrected input vector and the calculation model of the neural network, the output vector of the first network layer to be processed is obtained. The embodiment of the present application adds the perturbation vector to the input vector of the first network layer to be processed, which can reduce the correlation between the output feature images, speed up the training convergence speed, and reduce the precision loss. Moreover, the embodiment of the present application can generate a perturbation vector with the same dimension as the input vector of the first network layer to be processed through the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed. Since the dimension of the random vector of the first network layer to be processed is smaller than the dimension of the input vector, only a very small amount of storage resources and computing resources need to be added to ensure that the accuracy and convergence speed of the neural network are effectively improved.

[0132] The present application also provides a neural network model training method for obtaining the weight parameters of each network layer in the neural network before the neural network calculation, such as Fig.10 As shown, the training method of the neural network model includes steps S1001-S1004.

[0133] S1001. Initialize the weight parameters of each network layer.

[0134] Exemplarily, the weight parameters of each network layer in one or more first network layers to be processed and one or more second network layers to be processed included in the neural network are initialized.

[0135] S1002. Perform neural network calculation according to the neural network calculation model corresponding to each network layer to obtain a temporary output vector of the neural network.

[0136] Among them, the weight parameters of the neural network calculation model are initialized weight parameters.

[0137] Exemplarily, the above-mentioned performing neural network calculation according to the neural network calculation model corresponding to each network layer may include:

[0138] For the first network layer to be processed in the first network layer set, its temporary output vector is: Y m0 =W m0 ×X′ m +b m , where Y m0 is the temporary output vector of the first network layer to be processed, X′m is the correction input vector of the first network layer to be processed, W m0 is the block structured matrix corresponding to the initialized weight parameters of the first network layer to be processed, b m is the preset bias value of the first network layer to be processed.

[0139] For the first to-be-processed network layer in the second network layer set, its temporary output vector is: Y n0 =W n0 ×X n +b n , where Y n0 is the temporary output vector of the first network layer to be processed in the second network layer set, X n is the input vector of the first network layer to be processed in the second network layer set, W n0 is the block structured matrix corresponding to the initialized weight parameters of the first to-be-processed network layer in the second network layer set, b n is a preset bias value of the first to-be-processed network layer in the second network layer set.

[0140] For the second network layer to be processed, its temporary output vector is: Y i0 =C i0 ×X i +b i , where Y i is the temporary output vector of the second network layer to be processed, X i is the input vector of the second network layer to be processed, C i is the initial weight parameter of the second network layer to be processed, b i is the preset bias value of the second network layer to be processed.

[0141] Exemplarily, when the multiple network layers of the neural network only include the first network to be processed, Figure 4 or Figure 5 The calculation method shown in the figure performs neural network calculation on each network layer; or, when the multiple network layers of the neural network only include the first network to be processed, it can also be based on Figure 4 or Figure 5 The calculation method shown, and Figure 7 The calculation method shown in the figure performs neural network calculation on the corresponding network layer; or, when the multiple network layers of the neural network include the first network layer to be processed and the second network layer to be processed, the calculation method can be performed according to Figure 4 or Figure 5 The calculation method shown, and Figure 8 The calculation method described above performs neural network calculation on the corresponding network layer; or, when the multiple network layers of the neural network include the first network layer to be processed and the second network layer to be processed, it is also possible to perform neural network calculation on the corresponding network layer according to Figure 4or Figure 5 The calculation method shown, and Figure 7 and Figure 8 The calculation method described above performs neural network calculation on the corresponding network layer. The specific calculation method used can be determined according to the network architecture of the actual application and is not limited here. It should be noted that the calculation method used by the embodiment of the present application for each network layer during the training process should be the same as the calculation method used in the neural network reasoning process.

[0142] S1003. Update the weight parameters of each network layer through the reverse transfer of the neural network.

[0143] Exemplarily, the updated weight parameters of each network layer are used as weight parameters of the neural network calculation model corresponding to each network layer, and steps S1002-S1003 are repeated until the difference between the temporary output vector of the neural network and the preset output vector of the neural network is less than a preset value.

[0144] Exemplarily, when the difference between the temporary output vector of the neural network and the preset output vector of the neural network is less than a preset value, the weight parameter of the neural network can make the training converge. The embodiment of the present application does not limit the value of the preset value, which can be determined according to actual application.

[0145] S1004. Obtain weight parameters of each network layer.

[0146] For example, the weight parameter when the neural network training converges is the trained weight parameter, and the weight parameter can be used to perform the above Figure 4-Figure 8 The neural network reasoning shown can achieve a larger compression ratio, allowing the neural network to converge quickly.

[0147] Exemplarily, the training method of the neural network model provided in the embodiment of the present application may further include steps S601-S603 before step S901. According to the reference random vector determined in steps S601-S603 and the dimension of the random vector of the first network layer to be processed, the model training method of steps S1001-S1004 is used to obtain the weight parameters of each network layer, and then the aforementioned neural network calculation process is performed.

[0148] The training method of the neural network model provided in the embodiment of the present application is by initializing the weight parameters of each network layer; performing neural network calculation according to the neural network calculation model corresponding to each network layer to obtain a temporary output vector of the neural network; updating the weight parameters of each network layer through the reverse transmission of the neural network, and using the updated weight parameters of each network layer as the weight parameters of the neural network calculation model corresponding to each network layer, repeating the calculation until the difference between the temporary output vector of the neural network and the preset output vector of the neural network is less than the preset value, and obtaining the weight parameters of each network layer. The training process of the embodiment of the present application obtains a corrected input vector by adding a perturbation vector to the input vector of the first network layer to be processed, and performs neural network training through the corrected input vector, which can reduce the correlation between the output feature images, accelerate the training convergence speed, and reduce the accuracy loss.

[0149] The above mainly introduces the solution provided by the embodiment of the present invention from the perspective of method steps. It is understandable that in order to realize the above functions, the computer includes a hardware structure and / or software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of a combination of hardware and computer software. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0150] The embodiment of the present application can divide the computer into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0151] In the case of dividing each functional module into corresponding functional modules, Fig.11 A possible structural diagram of the neural network computing device involved in the above embodiment is shown, and the neural network computing device 1100 includes: a first acquisition unit 1101, a second acquisition unit 1102, a first computing unit 1103, a second computing unit 1104 and a decompression unit 1105. The first acquisition unit 1101 is used to support the neural network computing device 1100 to execute Figure 4 S401 in, or Figure 7 S701 in, or Figure 8 S801 in; the second acquisition unit 1102 is used to support the neural network computing device 1100 to execute Figure 4S402 in; the first computing unit 1103 is used to support the neural network computing device 1100 to execute Figure 4 S403 in; the second computing unit 1104 is used to support the neural network computing device 1100 to execute Figure 4 S404 in, or Figure 7 S703 in, or Figure 8 S802 in; decompression unit 1105 is used to support the neural network computing device 1100 to execute Figure 5 S405 in, or Figure 7 S702 in the above method embodiment. All relevant contents of each step involved in the above method embodiment can be cited in Figure 1 or Figure 2 The functional description of the corresponding functional modules shown will not be repeated here.

[0152] In the case of dividing each functional module into corresponding functional modules, Fig.12 A possible structural diagram of the neural network model training device involved in the above embodiment is shown, and the neural network model training device 1200 includes: an initialization unit 1201, a first determination unit 1202, a second determination unit 1203, a generation unit 1204, a neural network calculation unit 1205, a reverse transfer unit 1206 and an acquisition unit 1207. The initialization unit 1201 is used to support the neural network model training device 1200 to execute Fig.10 S1001 in; the first determining unit 1202 is used to support the neural network model training device 1200 to execute Figure 6 S601 in; the second determining unit 1203 is used to support the neural network model training device 1200 to execute Figure 6 S602 in; the generating unit 1204 is used to support the neural network model training device 1200 to execute Figure 6 S603 in; the neural network computing unit 1205 is used to support the neural network model training device 1200 to execute Fig.10 S1002 in; the reverse transfer unit 1206 is used to support the neural network model training device 1200 to execute Fig.10 S1003 in; the acquisition unit 1207 is used to support the neural network model training device 1200 to execute Fig.10 S1004 in.

[0153] In the case of an integrated unit, Fig.13A possible structural diagram of the neural network computing device 1300 involved in the above embodiment is shown. The neural network computing device 1300 includes: a storage module 1301 and a processing module 1302. The processing module 1302 is used to control and manage the actions of the computer. For example, the processing module 1302 is used to support the neural network computing device 1300 to execute Figure 4 S401-S404 in, or Figure 5 S401-S405 in, or Figure 6 S601-S603 in, or Figure 7 S701-S703 in, or Figure 8 S801-S802 in, and / or other processes for the technology described herein. Storage module 1301, for storing computer program code and data. In another implementation, the structure of the neural network computing device involved in the above embodiment can also include a processor and an interface, the processor and the interface communicate, and the processor is used to execute the embodiment of the present invention. The processor can be a CPU, or other hardware, such as a Field-Programmable Gate Array (FPGA), etc., or a combination of the two.

[0154] In the case of an integrated unit, Fig.14 A possible structural diagram of the neural network model training device 1400 involved in the above embodiment is shown. The neural network model training device 1400 includes: a storage module 1401 and a processing module 1402. The processing module 1402 is used to control and manage the actions of the neural network model training device 1400. For example, the processing module 1402 is used to support the neural network model training device 1400 to execute Figure 6 S601-S703 in, or Fig.10 S1001-S1004 in, and / or other processes for the technology described herein. Storage module 1401, for storing computer program code and data. In another implementation, the structure of the neural network model training device involved in the above embodiment can also include a processor and an interface, the processor and the interface communicate, and the processor is used to execute the embodiment of the present invention. The processor can be a CPU, or other hardware, such as a Field-Programmable Gate Array (FPGA), etc., or a combination of the two.

[0155] The steps of the method or algorithm described in conjunction with the disclosure of the present application may be implemented in hardware or by executing software instructions by a processor. The software instructions may be composed of corresponding software modules, which may be stored in a random access memory (RAM), a flash memory, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may be located in an ASIC. In addition, the ASIC may be located in a core network interface device. Of course, the processor and the storage medium may also exist in a core network interface device as discrete components.

[0156] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention may be implemented in hardware, software, firmware, or any combination thereof. When implemented using software, the functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium may be any available medium that a general or special-purpose computer can access.

[0157] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A neural network calculation method, characterized in that: The neural network includes a plurality of network layers, wherein the plurality of network layers includes a first network layer to be processed, and the method includes: Obtaining an input vector of the first network layer to be processed; the input vector includes pixel values ​​of an input image; the input image includes an image input to the neural network or a feature image output by a previous network layer of the first network layer to be processed; Acquire a perturbation vector of the first network layer to be processed according to the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed, wherein the dimension of the perturbation vector is equal to the dimension of the input vector of the first network layer to be processed, and the dimension of the random vector of the first network layer to be processed is determined based on the weight parameter of the first network layer to be processed and the dimension of the input vector of the first network layer to be processed; Multiplying the elements in the input vector of the first network layer to be processed by the elements at corresponding positions in the perturbation vector to obtain a corrected input vector of the first network layer to be processed; the corrected input vector includes the pixel values ​​of the input image after adding the perturbation vector; Based on the corrected input vector and the computational model of the neural network, an output vector of the first network layer to be processed is obtained; the output vector is a feature image of the input image.

2. The neural network calculation method according to claim 1, characterized in that: When the first network layer to be processed is the input layer of the neural network, obtaining the input vector of the first network layer to be processed includes using the image input to the neural network as the input vector of the first network layer to be processed; when the first network layer to be processed is not the input layer of the neural network, obtaining the input vector of the first network layer to be processed includes using the feature image of the output of the previous network layer of the first network layer to be processed as the input vector of the first network layer to be processed.

3. The neural network calculation method according to claim 1, characterized in that: The input vector of the first network layer to be processed is , the perturbation vector is , the correction input vector of the first network layer to be processed is , where n is a positive integer.

4. The neural network calculation method according to claim 1, characterized in that: The neural network calculation model includes: ,in, is the output vector of the neural network calculation model, is the input vector of the neural network calculation model, Calculate the weight parameters of the neural network model, is a preset bias value of the neural network calculation model, and the neural network calculation model is used for the neural network calculation of the jth layer among the multiple network layers, where j is an integer.

5. The neural network calculation method according to claim 4, characterized in that: The step of obtaining the output vector of the first to-be-processed network layer based on the correction input vector and the computing model of the neural network includes: The weight parameters and the corrected input vector of the first network layer to be processed are respectively used as the weight parameters and the input vector of the neural network calculation model to calculate to obtain the output vector of the first network layer to be processed.

6. The neural network calculation method according to claim 4 or 5, characterized in that: The weight parameter is a compression matrix corresponding to the block structured matrix, wherein the block structured matrix is ​​uniquely determined by the compression matrix, and before the output vector of the first to-be-processed network layer is obtained by performing a neural network calculation based on the corrected input vector, the method further includes: The weight parameters are decompressed into a block-structured matrix corresponding to the compressed matrix.

7. The neural network calculation method according to claim 6, characterized in that: Determining the dimension of the random vector of the first to-be-processed network layer based on the weight parameter of the first to-be-processed network layer and the dimension of the input vector of the first to-be-processed network layer includes: Determine a target interval, wherein two interval endpoints of the target interval are respectively the block size of the block structured matrix corresponding to the weight parameter of the first network layer to be processed and the dimension of the input vector of the first network layer to be processed; The dimension of the random vector of the first to-be-processed network layer is randomly determined within the target interval.

8. The neural network calculation method according to claim 1, characterized in that: The multiple network layers include multiple first network layers to be processed, and before obtaining the perturbation vector of the first network layer to be processed according to the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed, the method further includes: Determine the maximum value among the dimensions of all the random vectors of the first to-be-processed network layers as the dimension of the reference random vector of the neural network; Based on a preset random number generation model, generating a random number that satisfies the dimension of the reference random vector; The generated random numbers are used to form the reference random vector.

9. The neural network calculation method according to claim 1, characterized in that: Acquiring a perturbation vector of the first to-be-processed network layer according to the reference random vector of the neural network and the dimension of the random vector of the first to-be-processed network layer includes: Intercepting a vector having the same dimension as the random vector of the first to-be-processed network layer from the reference random vector as the random vector of the first to-be-processed network layer; The random vector of the first network layer to be processed is subjected to cyclic shift or vector element replication to generate a vector having the same dimension as the input vector of the first network layer to be processed as a disturbance vector of the first network layer to be processed.

10. A training method for a neural network model, for obtaining weight parameters of each network layer in a neural network, characterized in that: The method comprises: Step 1: Initialize the weight parameters of each network layer; Step 2: performing neural network calculation according to the neural network calculation model corresponding to each network layer to obtain a temporary output vector of the neural network, wherein the temporary output vector is a feature image; the weight parameter of the neural network calculation model is the initialized weight parameter, and performing neural network calculation according to the neural network calculation model corresponding to each network layer comprises performing neural network calculation on one or more of the first to-be-processed network layers in each network layer according to the neural network calculation method according to any one of claims 1 to 9; Step 3: Update the weight parameters of each network layer through the reverse transfer of the neural network; Using the updated weight parameters of each network layer as the weight parameters of the neural network calculation model corresponding to each network layer, repeating steps 2 and 3 until the difference between the temporary output vector and the preset output vector of the neural network is less than a preset value; Step 4: Obtain the weight parameters of each network layer.

11. A neural network computing device, characterized in that: The neural network includes a plurality of network layers, wherein the plurality of network layers includes a first network layer to be processed, and the device includes: A first acquisition unit is used to acquire an input vector of the first network layer to be processed; the input vector includes a pixel value of an input image; the input image includes an image input to the neural network or a feature image output by a previous network layer of the first network layer to be processed; A second acquisition unit is used to acquire a perturbation vector of the first network layer to be processed according to the reference random vector of the neural network and the dimension of the random vector of the first network layer to be processed, wherein the dimension of the perturbation vector is equal to the dimension of the input vector of the first network layer to be processed, and the dimension of the random vector of the first network layer to be processed is determined according to the weight parameter of the first network layer to be processed and the dimension of the input vector of the first network layer to be processed; A first calculation unit is used to multiply an element in the input vector of the first network layer to be processed by an element at a corresponding position in the disturbance vector obtained by the second acquisition unit to obtain a corrected input vector of the first network layer to be processed; the corrected input vector includes a pixel value of the input image after adding the disturbance vector; The second computing unit is used to obtain the output vector of the first network layer to be processed based on the corrected input vector obtained by the first computing unit and the computing model of the neural network; the output vector is a feature image of the input image.

12. The neural network computing device according to claim 11, characterized in that: When the first network layer to be processed is the input layer of the neural network, the acquisition unit is specifically used to acquire the image input into the neural network; when the first network layer to be processed is not the input layer of the neural network, the acquisition unit is specifically used to acquire the feature image of the output of the previous network layer of the first network layer to be processed.

13. The neural network computing device according to claim 11, characterized in that: The input vector of the first network layer to be processed is , the perturbation vector is , the correction input vector of the first network layer to be processed is , where n is a positive integer.

14. The neural network computing device according to claim 11, characterized in that: The neural network calculation model includes: ,in, is the output vector of the neural network calculation model, is the input vector of the neural network calculation model, Calculate the weight parameters of the neural network model, is a preset bias value of the neural network calculation model, and the neural network calculation model is used for the neural network calculation of the jth layer among the multiple network layers, where j is an integer.

15. The neural network computing device according to claim 14, characterized in that: The second calculation unit is specifically used to: calculate the weight parameters and the corrected input vector of the first network layer to be processed as the weight parameters and the input vector of the neural network calculation model respectively, so as to obtain the output vector of the first network layer to be processed.

16. The neural network computing device according to claim 14 or 15, characterized in that: The weight parameter is a compression matrix corresponding to the block structured matrix, wherein the block structured matrix is ​​uniquely determined by the compression matrix, and the device further includes a decompression unit, The decompression unit is used to decompress the weight parameter into a block-structured matrix corresponding to the compressed matrix.

17. The neural network computing device according to claim 16, characterized in that: The device further comprises a first determining unit, The first determining unit is used to determine a target interval, wherein two interval endpoints of the target interval are respectively a block size of a block structured matrix corresponding to a weight parameter of the first network layer to be processed and a dimension of an input vector of the first network layer to be processed; The first determining unit is further configured to randomly determine the dimension of the random vector of the first to-be-processed network layer within the target interval.

18. The neural network computing device according to claim 11, characterized in that: The multiple network layers include multiple first to-be-processed network layers, and the device further includes: a second determining unit and a generating unit, The second determining unit is further used to determine the maximum value among the dimensions of the random vectors of all the first to-be-processed network layers as the dimension of the reference random vector of the neural network; The generating unit is used to generate random numbers satisfying the dimension of the reference random vector based on a preset random number generating model, and to form the reference random vector with the generated random numbers.

19. The neural network computing device according to claim 11, characterized in that: The second acquisition unit is specifically configured to intercept a vector having the same dimension as the random vector of the first to-be-processed network layer from the reference random vector as the random vector of the first to-be-processed network layer; The second acquisition unit is specifically configured to generate a vector having the same dimension as the input vector of the first network layer to be processed by cyclic shift or vector element replication for the random vector of the first network layer to be processed as a disturbance vector of the first network layer to be processed.

20. A training device for a neural network model, used to obtain weight parameters of each network layer in a neural network, characterized in that: The device comprises: An initialization unit, used to initialize the weight parameters of each network layer; A neural network calculation unit, configured to perform neural network calculation according to the neural network calculation model corresponding to each network layer to obtain a temporary output vector of the neural network, wherein the temporary output vector is a feature image; and the weight parameter of the neural network calculation model is the initialized weight parameter; The neural network calculation unit is specifically used to perform neural network calculation on one or more of the first to-be-processed network layers in each network layer according to the neural network calculation method according to any one of claims 1 to 9; A reverse transfer unit, used for updating the weight parameters of each network layer through reverse transfer of the neural network; An acquisition unit is used to acquire the weight parameters of each network layer, and the weight parameters of each network layer are acquired when the difference between the temporary output vector of the neural network and the preset output vector of the neural network is less than a preset value.

21. A computer storage medium having computer program code stored therein, characterized in that: When the computer program code runs on a processor, the processor executes the neural network calculation method as described in any one of claims 1 to 9, or executes the training method of the neural network model as described in claim 10.

Images