Sparse training method and processing system of neural network model

Abstract

The invention provides a sparse method, a training method and a processing system for a neural network model, and the sparse method comprises the steps: constructing a to-be-sparse neural network model as a directed acyclic graph (DAG), enabling each layer of input of the to-be-sparse neural network model to correspond to a node in the DAG, and enabling the to-be-sparse neural network model to be a sparse neural network model; the sparse neural network model can correspond to directed paths between two nodes of the DAG by a sparse mode of a sparse network layer, and the number of the directed paths is multiple; determining a target sub-graph from the DAG through a mask parameter, wherein the mask parameter is used for selecting a directed path between any two nodes of the DAG; and performing sparse operation on the to-be-sparse neural network model according to the sparse structure corresponding to the target sub-graph to obtain a sparse neural network model. The technical problem that the precision of the neural network model is reduced due to the adoption of a single sparse mode in the prior art is solved, and the technical effect of improving the precision of the neural network model is achieved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a sparse neural network model, a training method, and a processing system. Background Art

[0002] Neural Networks (NN) is a complex network system formed by a large number of simple processing units (called neurons) that are widely interconnected. It reflects many basic characteristics of human brain functions and is a highly complex nonlinear dynamic learning system. Neural networks have large-scale parallel, distributed storage and processing, self-organization, self-adaptation and self-learning capabilities, and are particularly suitable for processing imprecise and fuzzy information processing problems that require considering many factors and conditions at the same time.

[0003] As the scale of neural network models continues to grow, the storage and computing resource requirements during training or inference increase. Therefore, most neural network models need to compress their parameter scale, that is, perform sparse processing on the neural network models. However, the current sparse method generally leads to a decrease in the accuracy of the neural network model. For this technical problem, no effective solution has been proposed in the relevant technology. Summary of the invention

[0004] The embodiments of the present application provide a sparse, training method, and processing system for a neural network model to solve one or more of the above-mentioned technical problems.

[0005] In the first aspect, an embodiment of the present application provides a sparse method for a neural network model, constructing the neural network model to be sparse as a directed acyclic graph DAG, wherein each layer input of the neural network model to be sparse corresponds to a node in the DAG, and the sparse pattern of the network layer of the neural network model to be sparse corresponds to a directed path between two nodes of the DAG, and the directed paths include multiple paths; determining a target subgraph from the DAG through a mask parameter, wherein the mask parameter is used to select a directed path between any two nodes of the DAG; performing a sparse operation on the neural network model to be sparse according to the sparse structure corresponding to the target subgraph to obtain a sparse neural network model.

[0006] In a second aspect, an embodiment of the present application provides a method for training a neural network model, obtaining a sparse neural network model determined by the sparse method of the above-mentioned neural network model; determining the loss value of the sparse neural network model after performing a specified task through a loss function; if the loss value is within a preset range, determining the sparse neural network model as a target neural network model; if the loss value is not within the preset range, training the sparse structure and the weight parameters after sparseness according to the sparse structure to obtain the target neural network model.

[0007] In a third aspect, an embodiment of the present application provides a processing system for a neural network model, comprising a sparse module and / or a training module, wherein the sparse module uses the sparse method of the above-mentioned neural network model to sparse the neural network model, and the training module uses the training method of the above-mentioned neural network model to train the sparse neural network model.

[0008] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0009] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, including computer instructions, which implement any of the methods described above when executed by a processor.

[0011] Compared with the related art, this application has the following advantages:

[0012] In an embodiment of the present application, the neural network model to be sparse is constructed as a directed acyclic graph DAG, wherein each layer input of the neural network model to be sparse corresponds to a node in the DAG, and the sparse pattern of the network layer of the neural network model to be sparse corresponds to a directed path between two nodes of the DAG, and the directed path includes multiple paths; a target subgraph is determined from the DAG through a mask parameter, wherein the mask parameter is used to select a directed path between any two nodes of the DAG; a sparse operation is performed on the neural network model to be sparse according to the sparse structure corresponding to the target subgraph to obtain a sparse neural network model. That is to say, in the process of sparsifying the neural network model, the embodiment of the present application no longer uses a single-mode sparse structure (for example, all network layers of the entire neural network model that can be sparse are sparsed using a 2:4 sparse mode), but selects a sparse mode from a plurality of sparse modes through a mask parameter based on the characteristics of each network layer (for example, assuming that the neural network model has three network layers that can be sparse, then the sparse modes of the three network layers are 1:4, 2:4, and 2:8, respectively), and then determines a mixed sparse structure. Compared with the sparse structure of a single mode, the sparse structure of the mixed mode is more targeted, thereby solving the technical problem in the related art that the use of a single sparse mode leads to a decrease in the accuracy of the neural network model, and achieving the technical effect of improving the accuracy of the neural network model.

[0013] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0015] Figure 1 A schematic diagram of a scenario of a sparse solution for a neural network model provided in an embodiment of the present application is shown;

[0016] Figure 2 A flowchart of a sparse method for a neural network model provided in an embodiment of the present application is shown;

[0017] Figure 3 A schematic diagram of a DAG construction method provided in an embodiment of the present application is shown;

[0018] Figure 4 A flowchart of a training method for a neural network model provided in an embodiment of the present application is shown;

[0019] Figure 5 A structural block diagram of a sparse device of a neural network model provided in an embodiment of the present application is shown;

[0020] Figure 6 A structural block diagram of a training device for a neural network model provided in an embodiment of the present application is shown;

[0021] Figure 7 A structural block diagram of a processing system of a neural network model provided in an embodiment of the present application is shown, and

[0022] Figure 8 A block diagram of an electronic device used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0024] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0025] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application, such as Figure 1 As shown, it may include at least a server 01 and a client 02. Specifically, in the embodiment of the present application, the server 01 may include an independently operated server, or a distributed server, or a server cluster composed of multiple servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN for short), and basic cloud computing services such as big data and artificial intelligence platforms. The server 01 may include a network communication unit, a processor, and a memory, etc. Specifically, the server 01 can be used to perform sparse processing on a neural network model. The client 02 may include physical devices such as smart phones, desktop computers, tablet computers, laptops, digital assistants, smart wearable devices, smart speakers, car terminals, smart TVs, etc., and may also include software running in physical devices, such as web pages provided to users by some service providers, or applications provided to users by these service providers. Specifically, the client 02 can be used to perform specified tasks (e.g., tasks such as face recognition, behavior recognition, and image segmentation) based on a server-sparse neural network model or a trained neural network model.

[0026] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0027] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0028] Fine-grained structured sparsity is an algorithm that compresses the network by regularizing the trimming of parameters. N:M fine-grained structured sparsity means that there are N non-zero values ​​fixed in M ​​consecutive weight values, and the remaining elements are all zero values. However, the current architecture only supports a 2:4 sparsity mode. Although it has been verified in multiple artificial intelligence (AI) tasks that most neural network models will not be affected in model performance after being sparsified by 2:4. However, with the introduction of higher sparsity ratios (for example, a sparse module that can achieve higher sparsity ratio calculations is designed, a processor supports up to 32x acceleration, and a product will support sparse modes with higher acceleration ratios such as 2:8 and 1:4), a single sparse mode will inevitably lead to a decrease in the accuracy of the neural network model. In order to alleviate the decrease in accuracy and make full use of hardware acceleration performance. The embodiment of the present application provides a sparse method for a neural network model, such as Figure 2 The flowchart of the sparse method of the neural network model of one embodiment of the present application is shown, and the method may include:

[0029] Step S202, construct the neural network model to be sparse as a directed acyclic graph DAG, wherein each layer input of the neural network model to be sparse corresponds to a node in the DAG, and the sparse pattern of the network layer of the neural network model to be sparse corresponds to the directed path between two nodes of the DAG, and the directed path includes multiple paths. It should be noted that the above-mentioned network layer that can be sparse can be determined according to the task to be performed by the neural network model, including but not limited to: convolution layer (for extracting image features), pooling layer (for reducing the dimension of data, compressing data and maintaining spatial invariance), fully connected layer (for converting feature layer into predicted output), recursive layer (for processing sequence data), decoding layer (for restoring encoded features to original data). For example, when performing task 1, the network layer that can be sparse can be a convolution layer and a fully connected layer, and when performing task 2, the network layer that can be sparse can be a convolution layer, a pooling layer, and a fully connected layer. In addition, it should be noted that the above-mentioned sparse pattern includes but is not limited to: 1:4, 2:4, 4:8, 2:8, 1:16.

[0030] Step S204, determining the target subgraph from the DAG through the mask parameter, wherein the mask parameter is used to select a directed path between any two nodes of the DAG. Optionally, the implementation of this step may include: determining a directed path between any two nodes of the DAG according to the mask parameter to obtain multiple directed paths; determining the target subgraph according to the directions of the multiple directed paths. For example, assuming that the sparsity patterns of the convolutional layer include 2:4, 4:8, which correspond to directed path 1 and directed path 2 in the above DAG, respectively, and the sparsity patterns of the fully connected layer include 2:8, 1:16, which correspond to directed path 3 and directed path 4 in the above DAG, respectively. Through the mask parameters, it can be determined that the sparse pattern of the convolution layer is 2:4, and the sparse pattern of the fully connected layer is 2:8. Then the above target subgraph can be a directed graph consisting of path 1 and path 3. Alternatively, through the mask parameters, it can be determined that the sparse pattern of the convolution layer is 2:4, and the sparse pattern of the fully connected layer is 1:16. Then the above target subgraph can be a directed graph consisting of path 1 and path 4.

[0031] Step S206, performing a sparse operation on the neural network model to be sparse according to the sparse structure corresponding to the target subgraph to obtain a sparse neural network model. It should be noted that the sparse patterns included in the above sparse structure may all be different, or some of the sparse patterns included in the sparse structure may be different. For example, assuming that the network layer that can be sparse includes 5 layers, network layer 1, network layer 2, network layer 3, network layer 4 and network layer 5, then the sparse structure can be 1:4, 2:4, 4:8, 2:8, 1:16, or it can be 2:4, 2:4, 2:4, 4:8, 1:16. Optionally, the implementation method of this step may include: according to the sparse structure corresponding to the target subgraph (for example, 1:4, 2:4, 4:8), the weight parameters of the sparse network layers (for example, network layer 1, network layer 2, network layer 3) are set, that is, low weights are cut off (for example, network layer 1 fixes 1 non-zero value in 4 consecutive weight values, and the remaining elements are all zero values, network layer 2 fixes 2 non-zero values ​​in 4 consecutive weight values, and the remaining elements are all zero values, network layer 3 fixes 4 non-zero values ​​in 8 consecutive weight values, and the remaining elements are all zero values), thereby obtaining a sparse neural network model.

[0032] Through the above steps S202 to S206, the neural network model to be sparse is constructed as a directed acyclic graph DAG, wherein each layer input of the neural network model to be sparse corresponds to a node in the DAG, and the sparse pattern of the network layer of the neural network model to be sparse corresponds to a directed path between two nodes of the DAG, and the directed path includes multiple paths; a target subgraph is determined from the DAG through a mask parameter, wherein the mask parameter is used to select a directed path between any two nodes of the DAG; a sparse operation is performed on the neural network model to be sparse according to the sparse structure corresponding to the target subgraph to obtain a sparse neural network model. That is to say, in the process of sparsifying the neural network model, the embodiment of the present application no longer uses a single-mode sparse structure (for example, all network layers of the entire neural network model that can be sparse are sparsed using a 2:4 sparse mode), but selects a sparse mode from a plurality of sparse modes through a mask parameter based on the characteristics of each network layer (for example, assuming that the neural network model has three network layers that can be sparse, then the sparse modes of the three network layers are 1:4, 2:4, and 2:8, respectively), and then determines a mixed sparse structure. Compared with the sparse structure of a single mode, the sparse structure of the mixed mode is more targeted, thereby solving the technical problem in the related art that the use of a single sparse mode leads to a decrease in the accuracy of the neural network model, and achieving the technical effect of improving the accuracy of the neural network model.

[0033] In a possible implementation, before determining the target subgraph from the DAG through the mask parameter, the mask parameter needs to be trained, wherein the training step may include: S11, defining the mask parameter using the Gumbel-Softmax function; S12, regulating the Gumbel-Softmax function through the variable parameter until the sum of all values ​​of the mask parameter is 1. It should be noted that the above-mentioned Gumbel-Softmax function is a special Softmax function, which allows the process of sampling from a discrete distribution in the neural network model to become differentiable, thereby allowing the neural network model parameters to be updated with gradients during back propagation. In the above S12, when the variable parameter tends to ∞, the above-mentioned mask parameter becomes a continuous random variable, and when the variable parameter tends to 0, the mask parameter becomes a discrete variable. During training, the variable parameter can be controlled to gradually become smaller, so that the distribution of the mask parameter will approach the one-hot encoding as the training progresses. In the vector of the one-hot code, only one position is 1, and the rest of the positions are 0. The mask parameters are trained through the above S11 to S12, and then a better mixed sparse structure can be obtained through the trained mask parameters.

[0034] In a possible implementation, constructing the neural network model to be sparse as a directed acyclic graph DAG includes: S21, each layer input of the neural network model to be sparse corresponds to a node, and multiple nodes are obtained; S22, setting the multiple nodes to nodes including the following information: the previous node of the current node, the weight parameters of multiple sparse modes between the current node and the previous node, and the multiple sparse modes between the current node and the previous node. Compared with the prior art traversing each possible sparse structure (for example, assuming that the number of layers that can be sparse is N, and the hardware supports K sparse modes, then all possible sparse structures have K N , it is necessary to N The above S21-S22 constructs the sparse neural network model to be sparse as DAG, which reduces the cost of training and further provides a basis for determining the global optimal sparse structure.

[0035] The following is an example of the sparse method of the neural network model provided in the embodiment of the present application.

[0036] Assume that the number of layers that can be sparse in a neural network model is N, and there are K types of sparse modes currently supported by hardware, then there are K possible sparse structures. N If we traverse each structure, the training cost will be very high. Therefore, we need an algorithm to Determine a globally optimal sparse structure a′.

[0037] like Figure 3 As shown, each layer of the neural network input is represented as a node v i ∈V, at node v i and v j There are K kinds of sparse operations in the middle, using edge Then v j It can be expressed as shown in formula (1).

[0038]

[0039] in, represents the weight parameter after sparseness in the kth sparse mode. The entire neural network is reconstructed into a directed acyclic graph G = (V, E), and any sparse structure can all be considered as a subgraph of graph G.

[0040] At any two nodes v i and v j There are k edges between them, but the sparse neural network can only select one of them in each layer operation. Therefore, this example introduces the mask And let vj The calculation of can be redefined as formula (2).

[0041]

[0042] Now we only need to get the sparse mask through training The optimal mixed sparse structure a′ can be determined. Since it is defined as a discrete variable, its gradient cannot be calculated in back propagation, so in this example, Gumbel-Softmax is used to define As shown in formula (3).

[0043]

[0044] Here represents the trainable parameters, is a random number that conforms to the Gumbel distribution. τ is a variable parameter to control the behavior of Gumbel-Softmax. When τ→∞, becomes a continuous random variable. When τ→0, Gradually becomes a discrete variable. During the training process, τ can be controlled to gradually become smaller, so that As training progresses, its distribution approaches one-hot. τ can be set as a variable related to the training epoch, as defined in formula (4).

[0045] τ=T0exp(-η×epoch) (4)

[0046] Among them, T0 represents temperature, η regulates the number of epoch training. One epoch is equal to the process of training once with all samples in the training set. Generally, T0 is set to 5.0, -η is set to 0.02, and epoch is set to 300.

[0047] In order to search for a mixed sparse structure that minimizes the loss of accuracy and computational cost of the neural network, the present application embodiment also provides a training method for a neural network model, such as Figure 4 The flowchart of the training method of the neural network model of one embodiment of the present application is shown, and the method may include:

[0048] Step S402, obtaining a sparse neural network model determined by the sparse method of the neural network model.

[0049] Step S404, determining the loss value of the sparse neural network model after performing the specified task through the loss function. It should be noted that the above-mentioned specified tasks may include image classification tasks, target detection tasks, image segmentation tasks, natural language processing, etc. Optionally, the above-mentioned loss function can be defined by the following information: the cross entropy loss of the sparse neural network model performing the specified task, the parameter storage cost loss and the computational cost loss when the sparse neural network model performs the specified task, wherein the cross entropy loss is used to describe the accuracy loss of the sparse neural network model performing the specified task. The determination method of the above-mentioned parameter storage cost loss may include: S31, determining the weight parameter amount and storage bit number of each network layer of the sparse neural network model; S32, determining the parameter storage cost loss through the weight parameter amount and the storage bit number; the determination method of the above-mentioned computational cost loss may include: S33, determining the computational amount of each network layer of the sparse neural network model and the computational efficiency of the hardware executing the sparse neural network model; S34, determining the computational cost loss through the computational amount and the computational efficiency.

[0050] Step S406, if the loss value is within a preset range, determining the sparse neural network model as a target neural network model;

[0051] Step S408, if the loss value is not within the preset range, the sparse structure and the weight parameters after sparseness according to the sparse structure are trained to obtain the target neural network model. Optionally, this step may include: determining other subgraphs in the DAG except the target, determining the weight parameters according to the sparse structures corresponding to the other subgraphs, and obtaining the output neural network model, or determining other weight parameters of the sparse structure corresponding to the target subgraph to obtain the output network model. For example, assuming that the initial sparse structure is 2:4, 2:8, after obtaining the neural network model through the sparse structure, it is determined that the loss value is not within the preset range, then the sparse structure can be determined to be 2:4, 1:4, or the position of the non-zero value of the weight parameters of 2:4, 2:8 can be reset.

[0052] In the above steps S402 to S408, the sparse neural network model determined by the sparse method of the above neural network model is obtained; the loss value of the sparse neural network model after performing the specified task is determined by the loss function; if the loss value is within the preset range, the sparse neural network model is determined as the target neural network model; if the loss value is not within the preset range, the sparse structure and the weight parameters after sparse structure are trained to obtain the target neural network model. That is, the sparse structure determined by the sparse method of the above neural network model and the weight parameters after sparse structure are trained in combination with the loss function, and then a globally optimal neural network sparse model can be obtained.

[0053] In order to adjust the above loss values ​​to the same order of magnitude, the training method of the above neural network model also proposes: S41, performing equalization processing on the loss value determined by the loss function, wherein the equalization processing method includes: performing a logarithmic operation on the loss value. For example, assuming there are two loss values, namely loss value 1 and loss value 2, the loss value 1 is 10000, and the loss value 2 is 10. Obviously, the loss value 1 and the loss value 2 are difficult to fit. Then, after using the logarithmic operation proposed in the embodiment of the present application, the two loss values ​​can be adjusted to the same order of magnitude.

[0054] In one possible implementation, training the sparse structure and the weight parameters after sparseness according to the sparse structure includes: S51, setting the loss value to an initial value; S52, training the sparse structure and the weight parameters after sparseness according to the sparse structure by stochastic gradient descent SGD. It should be noted that the above SGD is an iterative optimization algorithm for minimizing the cost function. The algorithm randomly selects a training sample in each iteration, and uses the sample to update the model parameters, and then repeats this process multiple times. Through S51 to S52, the optimal sparse structure can be searched in the global sparse structure, and no additional cost will be added during the model deployment to the hardware.

[0055] The following is an example of the training method of the neural network model provided in the embodiment of the present application.

[0056] This example mainly searches for a mixed sparse structure that minimizes the loss of accuracy and computational cost of the neural network. The loss function is defined as shown in formula (5).

[0057]

[0058] Above is a certain hybrid sparse structure, w a Represents the weight parameter after sparseness. The loss function consists of two parts. CrossEntropy(a) represents the cross entropy loss, which is used to constrain the accuracy loss during training. Cost(a) represents the storage cost and computational cost loss, and is defined as shown in formula (6), formula (7), and formula (8).

[0059]

[0060]

[0061]

[0062] Here, SparseMode(·) indicates sparse operation on weight parameters, and Bit(·) indicates the number of bits for storing parameters. represents the weight parameter of the i-th and j-th layers, represents the computational amount of the i-th and j-th layers, Indicates the hardware computing efficiency. In order to balance the impact of the loss, the results need to be The operation is shown in formula (9).

[0063]

[0064] Here, β and α can adjust the impact of Cost(a) on the overall loss.

[0065] In this example, the stochastic gradient descent (SGD) method is used to train the sparsification parameter θ and the sparse weight parameter W. The solution of θ is shown in formula (10).

[0066]

[0067] Here w′ a represents the network parameters after sparseness. Here a represents a sparse result, that is, a flow chart in DAG, p θ represents the probability of the trainable sparsification parameter θ after training, m a It represents the conditional distribution probability of the trainable sparsification parameter θ in the Gumbel-Softmax function, and g is a random parameter that conforms to the Gumbel distribution. Its value is between 0 and 1.

[0068] In summary, this example provides a neural network model training method. Compared with the single-mode sparse method, mixed sparse can better balance accuracy and computational cost, and improve the network compression ratio while maintaining accuracy.

[0069] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a sparse device for a neural network model. Figure 5 The figure shows a block diagram of a sparse device structure of a neural network model according to an embodiment of the present application. The device may include:

[0070] A construction module 501 is used to construct the neural network model to be sparse as a directed acyclic graph DAG, wherein each layer input of the neural network model to be sparse corresponds to a node in the DAG, and the sparse pattern of the network layer of the neural network model to be sparse corresponds to a directed path between two nodes of the DAG, and the directed path includes multiple paths;

[0071] A first determination module 502, configured to determine a target subgraph from the DAG by using a mask parameter, wherein the mask parameter is used to select a directed path between any two nodes of the DAG;

[0072] The first acquisition module 503 is used to perform a sparse operation on the neural network model to be sparse according to the sparse structure corresponding to the target subgraph to obtain a sparse neural network model.

[0073] In one possible implementation, the device further includes: a processing module, configured to define the mask parameter using a Gumbel-Softmax function before determining the target subgraph from the DAG using the mask parameter; and to control the Gumbel-Softmax function using a variable parameter until the sum of all values ​​of the mask parameter is 1.

[0074] The first determination module 502 includes: a first determination unit, configured to determine a directed path between any two nodes of the DAG according to the mask parameter to obtain multiple directed paths; and a second determination unit, configured to determine the target subgraph according to the directions of the multiple directed paths.

[0075] The above-mentioned construction module 501 includes: a third determination unit, used to input each layer of the neural network model to be sparse into a corresponding node to obtain multiple nodes; a setting unit, used to set the multiple nodes to be nodes including the following information: the previous node of the current node, the weight parameters under multiple sparse modes between the current node and the previous node, and the multiple sparse modes between the current node and the previous node.

[0076] In a possible implementation, all sparse patterns included in the sparse structure are different, or some sparse patterns included in the sparse structure are different, wherein the sparse patterns include: 1:4, 2:4, 4:8, 2:8, 1:16.

[0077] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a training device for a neural network model. Figure 6 The figure shows a structural block diagram of a training device for a neural network model according to an embodiment of the present application. The device may include:

[0078] The second acquisition module 601 is used to obtain the sparse neural network model determined by the sparse device of the above-mentioned neural network model; the second determination module 602 is used to determine the loss value of the sparse neural network model after performing a specified task through a loss function; the third determination module 603 is used to determine the sparse neural network model as the target neural network model if the loss value is within a preset range; the training module 604 is used to train the sparse structure and the weight parameters after sparse structure if the loss value is not within the preset range to obtain the target neural network model.

[0079] In a possible implementation, the device further includes a fourth determination module, which is used to define the loss function by the following information before determining the loss value of the sparse neural network model after executing the specified task through the loss function: the cross entropy loss of the sparse neural network model when executing the specified task, the parameter storage cost loss and the computational cost loss when the sparse neural network model executes the specified task, wherein the cross entropy loss is used to describe the accuracy loss of the sparse neural network model when executing the specified task. The method for determining the parameter storage cost loss includes: determining the weight parameter amount and the number of storage bits of each network layer of the sparse neural network model; determining the parameter storage cost loss through the weight parameter amount and the number of storage bits; determining the computational cost loss includes: determining the computational amount of each network layer of the sparse neural network model and the computational efficiency of the hardware executing the sparse neural network model; determining the computational cost loss through the computational amount and the computational efficiency.

[0080] In a possible implementation, the apparatus further includes an equalization module for performing equalization processing on the loss value determined by the loss function, wherein the equalization processing method includes: performing a logarithmic operation on the loss value.

[0081] In a possible implementation, the training module 604 is also used to set the loss value as an initial value; and train the sparse structure and the weight parameters after sparseness according to the sparse structure by stochastic gradient descent (SGD).

[0082] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a processing system for a neural network model. Figure 7 The figure shows a structural block diagram of a processing system of a neural network model according to an embodiment of the present application. The system may include:

[0083] The sparse module 701 is used to sparse the neural network model using the method described above for sparse neural network model.

[0084] The training module 702 is used to train the sparse neural network model using the training method of the neural network model.

[0085] The functions of each module in each device and system in the embodiment of the present application can be referred to the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0086] Figure 8 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 8As shown, the electronic device includes: a memory 801 and a processor 802. The memory 801 stores a computer program that can be run on the processor 802. When the processor 802 executes the computer program, the method in the above embodiment is implemented. The number of the memory 801 and the processor 802 can be one or more.

[0087] The electronic device also includes:

[0088] The communication interface 803 is used to communicate with external devices and perform data exchange transmission.

[0089] If the memory 801, the processor 802 and the communication interface 803 are implemented independently, the memory 801, the processor 802 and the communication interface 803 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0090] Optionally, in a specific implementation, if the memory 801, the processor 802 and the communication interface 803 are integrated on a chip, the memory 801, the processor 802 and the communication interface 803 can communicate with each other through an internal interface.

[0091] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0092] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0093] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0094] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0095] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0096] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0097] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0098] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0099] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0100] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0101] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0102] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0103] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A sparse method for a neural network model, comprising: The neural network model to be sparse is constructed as a directed acyclic graph DAG, wherein each layer input of the neural network model to be sparse corresponds to a node in the DAG, and the sparse pattern of the network layer of the neural network model to be sparse corresponds to a directed path between two nodes of the DAG, and the directed path includes multiple paths; Determining a target subgraph from the DAG by using a mask parameter, wherein the mask parameter is used to select a directed path between any two nodes of the DAG; A sparse operation is performed on the neural network model to be sparse according to the sparse structure corresponding to the target subgraph to obtain a sparse neural network model.

2. The method according to claim 1, wherein: Before determining the target subgraph from the DAG by using the mask parameter, the method further includes: The mask parameters are defined using the Gumbel-Softmax function; The Gumbel-Softmax function is regulated by a variable parameter until the sum of all values ​​of the mask parameter is 1.

3. The method according to claim 2, wherein: Determining a target subgraph from the DAG according to the mask parameter includes: Determine a directed path between any two nodes of the DAG according to the mask parameter to obtain multiple directed paths; The target subgraph is determined according to the directions of the multiple directed paths.

4. The method according to claim 1, wherein: The step of constructing the neural network model to be sparse as a directed acyclic graph DAG includes: Inputting each layer of the neural network model to be sparse corresponds to a node to obtain multiple nodes; The multiple nodes are set to be nodes including the following information: the previous node of the current node, the weight parameters in multiple sparse modes between the current node and the previous node, and the multiple sparse modes between the current node and the previous node.

5. The method according to claim 1, wherein: The sparse patterns included in the sparse structure are all different, or some of the sparse patterns included in the sparse structure are different, wherein the sparse patterns include: 1:4, 2:4, 4:8, 2:8, 1:

16.

6. A method for training a neural network model, comprising: Obtaining a sparse neural network model determined by the method according to any one of claims 1 to 5; Determine the loss value of the sparse neural network model after performing a specified task through a loss function; If the loss value is within a preset range, the sparse neural network model is determined as a target neural network model; If the loss value is not within a preset range, the sparse structure and the weight parameters after sparseness according to the sparse structure are trained to obtain a target neural network model.

7. The method according to claim 6, wherein: Before determining the loss value of the sparse neural network model after performing the specified task through the loss function, the method further includes: defining the loss function through the following information: The cross entropy loss of the sparse neural network model when performing the specified task, the parameter storage cost loss and the computational cost loss when the sparse neural network model performs the specified task, wherein the cross entropy loss is used to describe the accuracy loss of the sparse neural network model when performing the specified task.

8. The method according to claim 7, wherein: The method for determining the parameter storage cost loss includes: determining the weight parameter amount and the number of storage bits of each network layer of the sparse neural network model; determining the parameter storage cost loss according to the weight parameter amount and the number of storage bits; The method for determining the computational cost loss includes: determining the computational amount of each network layer of the sparse neural network model and the computational efficiency of hardware executing the sparse neural network model; and determining the computational cost loss based on the computational amount and the computational efficiency.

9. The method according to claim 8, wherein: The method further comprises: The loss value determined by the loss function is subjected to equalization processing, wherein the equalization processing method includes: performing a logarithmic operation on the loss value.

10. The method according to claim 9, wherein: The training of the sparse structure and the weight parameters after sparseness according to the sparse structure include: Setting the loss value to an initial value; The sparse structure and the weight parameters after sparseness according to the sparse structure are trained by stochastic gradient descent (SGD).

11. A processing system for a neural network model, comprising a sparse module and / or a training module, wherein: The sparse module uses the method described in any one of claims 1-5 to sparse the neural network model, and the training module uses the method described in any one of claims 6-10 to train the sparse neural network model.

12. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 5 or any one of claims 6 to 10 when executing the computer program.

13. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 or any one of claims 6 to 10 is implemented.

14. A computer program product, comprising computer instructions, wherein when the computer instructions are executed by a processor, the method according to any one of claims 1 to 5 or any one of claims 6 to 10 is implemented.

Images