Neural network model processing method and device, data processing method and device

Abstract

The present application discloses a method and device for processing a neural network model, and a method and device for processing data. The method for processing the neural network model includes: training the neural network model using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model; obtaining a target mask matrix corresponding to the pre-trained model, wherein the dimension of the target mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model; and processing the weight matrix of the pre-trained model using the target mask matrix to obtain a target model. The present application solves the technical problem in the related art that the computational complexity of the neural network model is large, which increases the hardware cost and power consumption.

Description

Technical Field

[0001] The present application relates to the field of machine learning, and more specifically, to a method and device for processing a neural network model, and a method and device for processing data. Background Art

[0002] The neural network model is an algorithmic mathematical model that can perform distributed parallel information processing. It relies on the complexity of the model and adjusts the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information. However, the higher the complexity of the neural network model, the greater the computing overhead and content storage consumption, resulting in a corresponding increase in hardware cost and power consumption.

[0003] Currently, no effective solution has been proposed to address the problem that neural network models in related technologies have large computational complexity, increased hardware costs and power consumption. Summary of the invention

[0004] The embodiments of the present application provide a method and device for processing a neural network model, and a method and device for processing data, so as to at least solve the technical problems in the related art that the neural network model has a large amount of calculation, and increases hardware costs and power consumption.

[0005] According to one aspect of an embodiment of the present application, a method for processing a neural network model is provided, including: training the neural network model using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model; obtaining a target mask matrix corresponding to the pre-trained model, wherein the dimension of the mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model; and processing the weight matrix of the pre-trained model using the mask matrix to obtain a target model.

[0006] According to another aspect of an embodiment of the present application, a processing device for a neural network model is also provided, including: a first training module, used to train the neural network model using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model; an acquisition module, used to obtain a target mask matrix corresponding to the pre-trained model, wherein the dimension of the mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model; a first processing module, used to process the weight matrix of the pre-trained model using the mask matrix to obtain a target model.

[0007] According to another aspect of an embodiment of the present application, a data processing method is also provided, including: obtaining a speech signal to be processed; processing the speech signal using a target model to obtain a recognition result, wherein the target model is obtained by processing a weight matrix of a pre-trained model using a target mask matrix, the pre-trained model is obtained by training a neural network model using a singular value decomposition algorithm and a norm regularization algorithm, and the dimension of the target mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model.

[0008] According to another aspect of an embodiment of the present application, a data processing device is also provided, including: an acquisition module for acquiring a speech signal to be processed; a processing module for recognizing the speech signal using a target model to obtain a recognition result, wherein the target model is obtained by processing a weight matrix of a pre-trained model using a target mask matrix, and the pre-trained model is obtained by training a neural network model using a singular value decomposition algorithm and a norm regularization algorithm, and the dimension of the target mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model.

[0009] According to another aspect of an embodiment of the present application, a storage medium is also provided, the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above-mentioned neural network model processing method.

[0010] According to another aspect of an embodiment of the present application, a computing device is also provided, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the above-mentioned processing method of the neural network model is executed when the program is run.

[0011] According to another aspect of an embodiment of the present application, a processing system for a neural network model is also provided, including: a processor; and a memory connected to the processor, for providing the processor with instructions for processing the following processing steps: training the neural network model using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model; obtaining a target mask matrix corresponding to the pre-trained model, wherein the dimension of the mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model; and processing the weight matrix of the pre-trained model using the mask matrix to obtain a target model.

[0012] In an embodiment of the present application, after obtaining the neural network model that needs to be compressed, the neural network model can be trained using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model, and the weight matrix of the pre-trained model can be further pruned using a set target mask matrix to achieve the purpose of maximally compressing the neural network model. Compared with the prior art, on the basis of SVD decomposition, the weight matrix is ​​continuously sparsely processed using a norm regularization algorithm, so that the sparsity of the weight matrix after SVD decomposition can be eliminated to the greatest extent, thereby achieving the technical effect of maximizing the reduction of the computational amount of the neural network model, further reducing the hardware cost and power consumption, and thus solving the technical problem of large computational amount of the neural network model in the related art, increasing hardware cost and power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0014] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing a processing method of a neural network model according to an embodiment of the present application;

[0015] Figure 2 is a flowchart of a method for processing a neural network model according to an embodiment of the present application;

[0016] Figure 3a is a flow chart of a training model according to an embodiment of the present application;

[0017] Figure 3b is a flow chart of a compression model according to an embodiment of the present application;

[0018] Figure 4 is a schematic diagram of a processing device for a neural network model according to an embodiment of the present application;

[0019] Figure 5 is a flow chart of a data processing method according to an embodiment of the present application; and

[0020] Figure 6 is a schematic diagram of a data processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0024] DNN: Deep Neural Networks, the internal neural network layers can be divided into three categories, namely input layer, hidden layer and output layer, and the layers are fully connected.

[0025] SVD: Singular Value Decomposition is a widely used algorithm in the field of machine learning. It can be used not only for feature decomposition in dimensionality reduction algorithms, but also for recommendation systems and natural language processing. It is the cornerstone of many machine learning algorithms. SVD can easily obtain the full rank decomposition of any matrix, and full rank decomposition can be used to compress data.

[0026] L21 norm: The L21 norm of matrix X can be defined as calculating the L2 norm of each column of data and then summing all the norms. The formula is as follows: Among them, x ij Represents the element in the i-th row and j-th column of matrix X.

[0027] Gradient truncation algorithm: The size of the gradient can be checked and limited during the training process. Check whether the value of the error gradient exceeds the threshold. If it exceeds, the gradient is truncated and the gradient is set to the threshold.

[0028] Example 1

[0029] According to an embodiment of the present application, a method for processing a neural network model is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a processing method of a neural network model is shown. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (BUS bus) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] The processor 102 may include the following three modules: a model training module, a model compression module and a data processing module, wherein the model training module may use a singular value decomposition algorithm and a norm regularization algorithm to train a neural network model, specifically including three parts: model modification, model training and coefficient adjustment; the model compression module may use a mask matrix of the same dimension to compress the weight matrix output by the model training module to obtain an optimized target model, specifically including three parts: mask matrix establishment, weight matrix compression and model retraining; the data processing module may use the model output by the model compression module to recognize the speech signal to obtain a recognition result.

[0032] It should be noted that, in some optional embodiments, the above Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the above-described computer device (or mobile device).

[0033] Under the above operating environment, this application provides Figure 2 The processing method of the neural network model shown. Figure 2 is a flowchart of a method for processing a neural network model according to an embodiment of the present application. Figure 2 Said method comprises the following steps:

[0034] Step S202, using a singular value decomposition algorithm and a norm regularization algorithm to train the neural network model to obtain a pre-trained model.

[0035] The neural network model in the above steps can be a convolutional neural network model, a recurrent neural network model, a deep neural network model, etc., but is not limited thereto. In the embodiment of the present application, a deep neural network DNN model is used as an example for illustration. The norm regularization algorithm can adopt the L21 norm regularization algorithm to achieve the purpose of sparse weight matrix by column.

[0036] It should be noted that in order to reduce the amount of calculation and memory-related indicators, and further reduce hardware costs, power consumption, and inference time, SVD can be used to prove that the following decomposition exists for any M*N matrix: This can be applied to data dimensionality reduction and compression. When the data correlation is particularly large, storing the X and Y matrices takes up less space than storing the A matrix. Assuming that the original matrix size is M*N, after SVD decomposition, it becomes two matrices M*A and A*N, so that the amount of calculation and storage are reduced. However, the matrix after SVD decomposition still has dilution.

[0037] In an optional embodiment, in order to achieve compression of the DNN model, it can be achieved through the cooperation of training and compression processes. During the training process, after performing SVD on the DNN model, the weight matrix can be structured sparse using the L21 norm.

[0038] Step S204, obtaining a target mask matrix corresponding to the pre-trained model, wherein the dimension of the target mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model.

[0039] The target mask matrix in the above steps can be a flag set for each column of weight values ​​in the weight matrix, which is used to indicate whether each column of weight values ​​can be compressed. For example, when a column of weight values ​​can be compressed, the corresponding element in the target mask matrix can be set to 0, and when a column of weight values ​​needs to be retained and cannot be compressed, the corresponding element in the target mask matrix can be set to 1.

[0040] In an optional embodiment, the value of each element in the target mask matrix can be set and adjusted by repeatedly training and compressing the pre-trained model until the accuracy and compression ratio of the compressed model can meet user requirements. At this time, the compressed model can meet user requirements, and the target mask matrix is ​​the mask matrix required for the final compression. It does not need to be adjusted again and can be used to prune the pre-trained model to achieve the purpose of compressing the model.

[0041] It should be noted that in the process of repeatedly compressing and training the pre-trained model, the target mask matrix is ​​only set based on the weight matrix of the compressed model, and the pre-trained model is not pruned.

[0042] Step S206, using the target mask matrix to process the weight matrix of the pre-trained model to obtain the target model.

[0043] Specifically, each hidden layer in the DNN model corresponds to a weight matrix, so the weight matrix in the above steps can be a weight matrix group.

[0044] In an optional embodiment, after obtaining the set mask matrix, the mask matrix can be used to perform a pruning operation on the weight matrix. The specific solution may be: all weight values ​​in the column corresponding to the 0 element in the mask matrix are pruned, while all weight values ​​in the column corresponding to the 1 element are retained, thereby achieving a compression of the DNN model by 65% ​​on the basis of SVD without losing the model accuracy.

[0045] Based on the solution provided by the above-mentioned embodiment of the present application, after obtaining the neural network model that needs to be compressed, the neural network model can be trained using the singular value decomposition algorithm and the norm regularization algorithm to obtain a pre-trained model, and the weight matrix of the pre-trained model can be further pruned using the set target mask matrix to achieve the purpose of compressing the neural network model to the maximum extent. Compared with the prior art, on the basis of SVD decomposition, the weight matrix is ​​continuously sparsely processed using the norm regularization algorithm, so that the sparsity of the weight matrix after SVD decomposition can be eliminated to the maximum extent, achieving the technical effect of maximizing the reduction of the computational amount of the neural network model, and further reducing the hardware cost and power consumption, thereby solving the technical problem of large computational amount of the neural network model in the related art, increasing hardware cost and power consumption.

[0046] In the above embodiment of the present application, the neural network model is trained using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model, including: modifying the neural network model using a singular value decomposition algorithm to obtain a modified network model; generating a loss function based on a norm regularization algorithm and a target coefficient; and training the modified network model using the loss function and the target coefficient to obtain a pre-trained model.

[0047] Specifically, the target coefficient in the above steps may be a sparse coefficient. By setting the coefficient, the accuracy can be reduced less while ensuring that the sparsity of the pre-trained model is as small as possible, which is within the range required by the user.

[0048] In an optional embodiment, the DNN network can be decomposed by SVD, and SVD network nodes can be added to obtain an SVD network (i.e., the modified network model mentioned above). Based on the SVD network, the L21 loss function of the column dimension is added for training, thereby pre-training the model so that the weight value of the entire column approaches 0 for sparseness.

[0049] In the above embodiment of the present application, before generating a loss function based on a norm regularization algorithm and a target coefficient, the method further includes: obtaining multiple sets of initial coefficients; training the modified network model using each set of initial coefficients to determine a first precision and a weight matrix corresponding to each set of initial coefficients; obtaining the sparsity of the weight matrix corresponding to each set of initial coefficients to obtain the matrix sparsity corresponding to each set of initial coefficients; and determining the target coefficient among the multiple sets of initial coefficients based on the first precision and the matrix sparsity corresponding to each set of initial coefficients.

[0050] Specifically, the multiple sets of initial coefficients in the above steps can be pre-set multiple sets of initial sparse coefficients, and the most suitable sparse coefficients can be determined by training and comparing the DNN model.

[0051] In an optional embodiment, multiple groups of different sparse coefficients can be pre-set, and the SVD network is trained using each group of initial sparse coefficients. According to the change in training accuracy after training and the sparse threshold required by the user, the sparse coefficients are automatically adjusted and the most suitable sparse coefficients are found to ensure that the accuracy meets the original reference accuracy, and the sparsity is smaller.

[0052] In the above embodiment of the present application, based on the first precision and matrix sparsity corresponding to each group of initial coefficients, determining the target coefficients in multiple groups of initial coefficients includes: comparing the first precision corresponding to each group of initial coefficients with the original precision, and comparing the matrix sparsity corresponding to each group of initial coefficients with the preset sparsity; if the first precision is within the preset range of the original precision and the matrix sparsity is less than the preset sparsity, then determining the corresponding initial coefficient as the target coefficient.

[0053] Specifically, the original accuracy in the above steps may be the accuracy of the DNN model, and the preset range may be a range that ensures that the accuracy meets the accuracy of the DNN model and is not worse than the accuracy. The preset sparsity in the above steps may be a sparsity threshold determined according to the user's compression requirements.

[0054] In an optional embodiment, the purpose of compressing the model is to reduce the amount of calculation, but it cannot affect the accuracy of the model. Therefore, the goal of pre-training should be to minimize the sparsity without affecting the accuracy of the DNN model. Based on this goal, multiple sets of pre-set sparse coefficients can be used for training, and the training accuracy change and matrix sparsity change after a period of training can be obtained. When the accuracy change is not worse than the accuracy of the DNN model, and the matrix sparsity is less than the preset sparsity, the current initial sparse coefficient can be determined as the target coefficient used for the final training.

[0055] In the above embodiment of the present application, before obtaining the target mask matrix corresponding to the pre-trained model, the method also includes: processing the pre-trained model using a norm regularization algorithm to determine the mask matrix; processing the weight matrix of the pre-trained model using the mask matrix to obtain a processed network model; retraining the processed network model using the loss function and the target coefficient to obtain a trained network model; determining whether the second precision of the trained network model meets the target precision; if the second precision does not meet the target precision, using the trained network model as the pre-trained model, and returning to execute the step of processing the pre-trained model using the norm regularization algorithm to determine the mask matrix until the second precision meets the target precision; if the second precision meets the target precision, determining the mask matrix as the target mask matrix.

[0056] Specifically, the target accuracy in the above steps may be the model accuracy required by the user.

[0057] In an optional embodiment, a mask matrix with the same dimensional size is established for the weight matrix to be optimized and compressed. For the weight matrix group of the DNN model, the L21 norm calculation can be performed on each weight matrix separately to determine the weight column that needs to be compressed, and the corresponding element value in the constructed mask matrix is ​​set to 0, and the other element values ​​are set to 1. The masked model is retrained to confirm whether the accuracy of the retrained model meets the user requirements. If not, the same loss function and the most suitable sparse coefficient as the pre-training are used for retraining, and the above steps are repeated; if it is satisfied, the mask obtained at this time is determined to be the final target mask matrix required.

[0058] In the above embodiment of the present application, the pre-trained model is processed using a norm regularization algorithm, and determining the mask matrix includes: performing norm processing on the weight matrix of the pre-trained model according to the grouping combination and column dimension to obtain norm values ​​of multiple weight columns; obtaining a weight column whose norm value is less than a first preset value to obtain a target weight column; and determining that the element value corresponding to the target weight column in the mask matrix is ​​a preset value.

[0059] Specifically, the first preset value in the above step may be a pre-set threshold for determining whether the weight column needs to be compressed. When the norm value is less than the threshold, it is determined that the weight column needs to be compressed; when the norm value is greater than or equal to the threshold, it is determined that the weight column does not need to be compressed. The preset value may be an element value in the mask matrix that determines that the weight column needs to be compressed, for example, it may be 0.

[0060] It should be noted that whether the weight column is compressed needs to be determined based on the norm value. Therefore, the values ​​of all elements in the established mask matrix can be 1. Each time it is determined that the weight column needs to be compressed, the value of the corresponding element is modified to 1.

[0061] In an optional embodiment, after each training is completed to obtain a trained model, L21 calculation can be performed according to the group combination, and the weight column with the smallest norm value calculated according to the column dimension is found, and the norm value is compared with the set threshold. If the norm value is less than the set threshold, it is determined that the weight column needs to be compressed, and the value of the element corresponding to the weight column in the mask matrix can be set to 0.

[0062] In the above embodiment of the present application, the pre-trained model is processed using a norm regularization algorithm, and determining the mask matrix includes: obtaining the mean of the absolute values ​​of the weight values ​​located in the same column in the weight matrix according to the column dimension to obtain a first mean of multiple weight columns; obtaining the mean of all weight values ​​in the weight matrix to obtain a second mean; obtaining the ratio of the first mean to the second mean of multiple weight columns to obtain a ratio of multiple weight columns; obtaining a weight column whose ratio is less than a second preset value to obtain a target weight column; and determining that the element value corresponding to the target weight column in the mask matrix is ​​a preset value.

[0063] Specifically, the second preset value in the above steps can be a pre-set threshold for determining whether the weight column needs to be compressed. When the ratio is less than the threshold, it is determined that the weight column needs to be compressed; when the ratio is greater than or equal to the threshold, it is determined that the weight column does not need to be compressed.

[0064] In an optional embodiment, for the weight matrix corresponding to each hidden layer, the mean of the absolute values ​​of all weights in each weight column is first calculated according to the column dimension to obtain a first mean, and then the average value of all weights in the weight matrix is ​​calculated to obtain a second mean. Finally, a global sorting is performed based on the ratio of the two means to screen out all weight columns whose means are less than a set threshold, determine that they need to be compressed, and set the values ​​of the corresponding elements in the mask matrix to 0.

[0065] In the above embodiment of the present application, in the process of retraining the processed network model using the loss function and the target coefficient, the method also includes: processing the processed network model using a gradient truncation algorithm.

[0066] In an optional embodiment, in order to prevent sparse weights from being updated with gradients, a gradient truncation algorithm may be used to process the weights in the weight matrix during model retraining.

[0067] Combine the following Figure 3a and 3b A preferred embodiment of the present application is described in detail.

[0068] Figure 3a A flow chart of a training model is shown, Figure 3a As shown, the method may include the following steps:

[0069] Step S31, perform SVD modification on the DNN network and add SVD network nodes.

[0070] Optionally, the DNN network in the above steps may be the above-mentioned neural network model.

[0071] Step S32: Based on the SVD network, L21 weight group training is superimposed.

[0072] Optionally, the L21 weight group in the above step can be the above-mentioned norm regularization algorithm. The weight group can be combined according to the group, and then the group L21 loss function with increased column dimension can be trained so that the entire column approaches 0, which is convenient for sparseness.

[0073] Step S33, based on multiple groups of initial sparse coefficients, according to the accuracy change after a period of training and the customized matrix sparse threshold, the sparse coefficients are automatically adjusted and the target sparse coefficients are found.

[0074] Optionally, the initial sparse coefficient in the above step may be the above initial coefficient, and the target sparse coefficient may be the above target coefficient. The screening target is that the accuracy is not much different from the original accuracy, and the sparsity is smaller.

[0075] Step S34, retraining is performed using the found target sparse coefficients to obtain a new accuracy.

[0076] Optionally, the SVD network is retrained using the found target sparse coefficients to obtain the final L21pretrain model, i.e., the pre-trained model mentioned above.

[0077] Figure 3b A flow chart of a compression model is shown, Figure 3b As shown, the method may include the following steps:

[0078] Step S35, creating a mask matrix of the same dimension size as the weight matrix to be compressed.

[0079] Optionally, the values ​​of all elements of the mask matrix are set to 1.

[0080] Step S36, perform L21 calculation on the weight group according to the group combination, and find the lowest weight column calculated according to the column dimension.

[0081] Step S37, calculate the mean of the absolute values ​​of all weights according to the column dimension, and the average value of all weights of the current layer, perform global sorting based on the ratio of the two means, and set the element value corresponding to the entire weight column in the mask matrix that is lower than the set threshold to 0.

[0082] Step S38, retraining the masked model, wherein the loss function and target sparse coefficient used in the L21 pretrain model training process are used for retraining.

[0083] Optionally, use a gradient phasing algorithm during training to prevent gradient updates for sparse weights.

[0084] Step S39, determining whether the accuracy and compression ratio meet the requirements.

[0085] Optionally, if the accuracy and compression ratio after retraining do not meet the requirements, continue to perform the above steps S36 to S39 until the accuracy and compression ratio meet the requirements.

[0086] Step S310: If the accuracy and compression ratio after retraining meet the requirements, the weight matrix is ​​compressed using the mask matrix to obtain the final optimized model.

[0087] Optionally, the optimization model in the above steps may be the above target model.

[0088] Through the above steps, on the basis of SVD matrix decomposition, the structured compression of L21 is superimposed to achieve the maximum structural sparseness of the weight matrix after SVD.

[0089] In the above embodiment of the present application, after using the target mask matrix to process the weight matrix of the pre-trained model to obtain the target model, the method also includes: obtaining a speech signal to be processed; and recognizing the speech signal using the target model to obtain a recognition result.

[0090] The voice signal to be processed in the above steps can be sent by the user and collected by an audio collection device such as a microphone.

[0091] In an optional embodiment, in the field of speech recognition, based on actual recognition needs, a target model can be trained in advance through the above solution, and the collected speech signal can be recognized using the target model to obtain relevant recognition results, and the device can be further controlled based on the recognition results, but it is not limited to this. Since the target model is an optimized and compressed model, the speech recognition speed is faster and the consumption is smaller.

[0092] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0094] Example 2

[0095] According to an embodiment of the present application, a processing device for a neural network model for implementing the processing method of the neural network model is also provided, such as Figure 4 As shown, the device 400 includes: a first training module 402 , an acquisition module 404 and a first processing module 406 .

[0096] Among them, the first training module 402 is used to train the neural network model using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model; the acquisition module 404 is used to obtain a target mask matrix corresponding to the pre-trained model, wherein the dimension of the mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model; the first processing module 406 is used to process the weight matrix of the pre-trained model using the mask matrix to obtain a target model.

[0097] It should be noted that the first training module 402, the acquisition module 404 and the first processing module 406 correspond to steps S202 to S206 in Example 1, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0098] Based on the solution provided by the above-mentioned embodiment of the present application, after obtaining the neural network model that needs to be compressed, the neural network model can be trained by the first training module using the singular value decomposition algorithm and the norm regularization algorithm to obtain a pre-trained model, and the weight matrix of the pre-trained model can be further pruned by the first processing module using the set target mask matrix to achieve the purpose of compressing the neural network model to the maximum extent. Compared with the prior art, on the basis of SVD decomposition, the weight matrix is ​​continuously sparsely processed using the norm regularization algorithm, so that the sparsity of the weight matrix after SVD decomposition can be eliminated to the maximum extent, achieving the technical effect of maximizing the reduction of the computational amount of the neural network model, and further reducing the hardware cost and power consumption, thereby solving the technical problem of large computational amount of the neural network model in the related art, increasing the hardware cost and power consumption.

[0099] In the above embodiments of the present application, Figure 4 As shown, the first training module 402 includes: a modification unit 408 , a generation unit 410 and a training unit 412 .

[0100] Among them, the modification unit 408 is used to modify the neural network model using the singular value decomposition algorithm to obtain a modified network model; the generation unit 410 is used to generate a loss function based on the norm regularization algorithm and the target coefficient; the training unit 412 is used to train the modified network model using the loss function and the target coefficient to obtain a pre-trained model.

[0101] In the above embodiments of the present application, Figure 4 As shown, the device 400 also includes: a determination module 414.

[0102] Among them, the acquisition module 404 is also used to obtain multiple groups of initial coefficients; the first training module 402 is also used to train the modified network model using each group of initial coefficients to determine the first precision and weight matrix corresponding to each group of initial coefficients; the acquisition module 404 is also used to obtain the sparsity of the weight matrix corresponding to each group of initial coefficients to obtain the matrix sparsity corresponding to each group of initial coefficients; the determination module 414 is used to determine the target coefficients among the multiple groups of initial coefficients based on the first precision and matrix sparsity corresponding to each group of initial coefficients.

[0103] In the above embodiments of the present application, Figure 4 As shown, the determination module 414 includes: a comparison unit 416 and a first determination unit 418 .

[0104] Among them, the comparison unit 416 is used to compare the first precision corresponding to each group of initial coefficients with the original precision, and to compare the matrix sparsity corresponding to each group of initial coefficients with the preset sparsity; the first determination unit 418 is used to determine that if the first precision is within the preset range of the original precision and the matrix sparsity is less than the preset sparsity, the corresponding initial coefficient is the target coefficient.

[0105] In the above embodiments of the present application, Figure 4 As shown, the device 400 also includes: a second processing module 420 , a third processing module 422 , a second training module 424 , a judgment module 426 and a return module 428 .

[0106] Among them, the second processing module 420 is used to process the pre-trained model using the norm regularization algorithm to determine the mask matrix; the third processing module 422 is used to process the weight matrix of the pre-trained model using the mask matrix to obtain the processed network model; the second training module 424 is used to retrain the processed network model using the loss function and the target coefficient to obtain the trained network model; the judgment module 426 is used to judge whether the second precision of the trained network model meets the target precision; the return module 428 is used to use the trained network model as the pre-trained model if the second precision does not meet the target precision, and return to execute the function of the second processing module until the second precision meets the target precision; the determination module 410 is used to determine that the mask matrix is ​​the target mask matrix if the second precision meets the target precision.

[0107] In the above embodiments of the present application, Figure 4 As shown, the second processing module 420 includes: a processing unit 430 , an acquiring unit 432 and a second determining unit 434 .

[0108] Among them, the processing unit 430 is used to perform norm processing on the weight matrix of the pre-trained model according to the grouping combination and column dimension to obtain the norm values ​​of multiple weight columns; the acquisition unit 432 is used to obtain the weight column whose norm value is less than the first preset value to obtain the target weight column; the second determination unit 434 is used to determine that the element value corresponding to the target weight column in the mask matrix is ​​the preset value.

[0109] In the above embodiment of the present application, the processing unit 430 is also used to obtain the mean of the absolute values ​​of the weight values ​​located in the same column in the weight matrix according to the column dimension, and obtain the first mean of multiple weight columns; the acquisition unit 432 is also used to obtain the mean of all weight values ​​in the weight matrix, and obtain the second mean; the acquisition unit 432 is also used to obtain the ratio of the first mean of multiple weight columns to the second mean, and obtain the ratio of multiple weight columns; the acquisition unit 432 is also used to obtain a weight column whose ratio is less than a second preset value, and obtain a target weight column; the second determination unit 434 is also used to determine that the element value corresponding to the target weight column in the mask matrix is ​​a preset value.

[0110] In the above embodiment of the present application, the second training module is also used to process the processed network model using a gradient truncation algorithm.

[0111] In the above embodiment of the present application, the device also includes: a fourth processing module.

[0112] Among them, the acquisition module is also used to acquire the voice signal to be processed; the fourth processing module is used to recognize the voice signal using the target model to obtain the recognition result.

[0113] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0114] Example 3

[0115] According to an embodiment of the present application, a data processing method is also provided.

[0116] Figure 5 is a flow chart of a data processing method according to an embodiment of the present application. Figure 5 Said method comprises the following steps:

[0117] Step S502: Acquire a speech signal to be processed.

[0118] The voice signal to be processed in the above steps can be sent by the user and collected by an audio collection device such as a microphone.

[0119] Step S504, using the target model to recognize the speech signal to obtain a recognition result, wherein the target model is obtained by processing the weight matrix of the pre-trained model using the target mask matrix, the pre-trained model is obtained by training the neural network model using the singular value decomposition algorithm and the norm regularization algorithm, and the dimension of the target mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model.

[0120] The neural network model in the above steps can be a convolutional neural network model, a recurrent neural network model, a deep neural network model, etc., but is not limited thereto. In the embodiments of the present application, a deep neural network DNN model is used as an example for illustration. The norm regularization algorithm can adopt the L21 norm regularization algorithm to achieve the purpose of sparse weight matrix by column. The target mask matrix can be a flag bit set for each column of weight values ​​in the weight matrix, which is used to indicate whether each column of weight values ​​can be compressed.

[0121] Specifically, each hidden layer in the DNN model corresponds to a weight matrix, so the weight matrix in the above steps can be a weight matrix group.

[0122] Based on the solution provided by the above-mentioned embodiment of the present application, after obtaining the neural network model that needs to be compressed, the singular value decomposition algorithm and the norm regularization algorithm can be used to train the neural network model to obtain a pre-trained model, and the weight matrix of the pre-trained model can be further pruned using the set target mask matrix, so that the acquired speech can be processed using the target model to obtain the recognition result, thereby achieving the purpose of data processing. Compared with the prior art, on the basis of SVD decomposition, the norm regularization algorithm is used to continuously sparse the weight matrix, so that the sparsity of the weight matrix after SVD decomposition can be eliminated to the greatest extent, achieving the technical effect of maximizing the reduction of the computational amount of the neural network model, and further reducing the hardware cost and power consumption, thereby solving the technical problem of large computational amount of the neural network model in the related technology, increasing hardware cost and power consumption.

[0123] In the above embodiment of the present application, training the neural network model using the singular value decomposition algorithm and the norm regularization algorithm includes: modifying the neural network model using the singular value decomposition algorithm to obtain a modified network model; generating a loss function based on the norm regularization algorithm and the target coefficient; and training the modified network model using the loss function and the target coefficient to obtain a pre-trained model.

[0124] In the above embodiment of the present application, before generating a loss function based on a norm regularization algorithm and a target coefficient, the method further includes: obtaining multiple sets of initial coefficients; training the modified network model using each set of initial coefficients to determine a first precision and a weight matrix corresponding to each set of initial coefficients; obtaining the sparsity of the weight matrix corresponding to each set of initial coefficients to obtain the matrix sparsity corresponding to each set of initial coefficients; and determining the target coefficient among the multiple sets of initial coefficients based on the first precision and the matrix sparsity corresponding to each set of initial coefficients.

[0125] In the above embodiment of the present application, based on the first precision and matrix sparsity corresponding to each group of initial coefficients, determining the target coefficients in multiple groups of initial coefficients includes: comparing the first precision corresponding to each group of initial coefficients with the original precision, and comparing the matrix sparsity corresponding to each group of initial coefficients with the preset sparsity; if the first precision is within the preset range of the original precision and the matrix sparsity is less than the preset sparsity, then determining the corresponding initial coefficient as the target coefficient.

[0126] In the above embodiment of the present application, before obtaining the target mask matrix corresponding to the pre-trained model, the method also includes: processing the pre-trained model using a norm regularization algorithm to determine the mask matrix; processing the weight matrix of the pre-trained model using the mask matrix to obtain a processed network model; retraining the processed network model using the loss function and the target coefficient to obtain a trained network model; determining whether the second precision of the trained network model meets the target precision; if the second precision does not meet the target precision, using the trained network model as the pre-trained model, and returning to execute the step of processing the pre-trained model using the norm regularization algorithm to determine the mask matrix until the second precision meets the target precision; if the second precision meets the target precision, determining the mask matrix as the target mask matrix.

[0127] In the above embodiment of the present application, the pre-trained model is processed using a norm regularization algorithm, and determining the mask matrix includes: performing norm processing on the weight matrix of the pre-trained model according to the grouping combination and column dimension to obtain norm values ​​of multiple weight columns; obtaining a weight column whose norm value is less than a first preset value to obtain a target weight column; and determining that the element value corresponding to the target weight column in the mask matrix is ​​a preset value.

[0128] In the above embodiment of the present application, the pre-trained model is processed using a norm regularization algorithm, and determining the mask matrix includes: obtaining the mean of the absolute values ​​of the weight values ​​located in the same column in the weight matrix according to the column dimension to obtain a first mean of multiple weight columns; obtaining the mean of all weight values ​​in the weight matrix to obtain a second mean; obtaining the ratio of the first mean to the second mean of multiple weight columns to obtain a ratio of multiple weight columns; obtaining a weight column whose ratio is less than a second preset value to obtain a target weight column; and determining that the element value corresponding to the target weight column in the mask matrix is ​​a preset value.

[0129] In the above embodiment of the present application, in the process of retraining the processed network model using the loss function and the target coefficient, the method also includes: processing the processed network model using a gradient truncation algorithm.

[0130] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0131] Example 4

[0132] According to an embodiment of the present application, a data processing device for implementing the above data processing method is also provided. Figure 6 As shown, the device 600 includes: an acquisition module 602 and a processing module 604 .

[0133] Among them, the acquisition module 602 is used to acquire the speech signal to be processed; the processing module 604 is used to use the target model to recognize the speech signal to obtain a recognition result, wherein the target model is obtained by processing the weight matrix of the pre-trained model using the target mask matrix, and the pre-trained model is obtained by training the neural network model using the singular value decomposition algorithm and the norm regularization algorithm, and the dimension of the target mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model.

[0134] It should be noted that the acquisition module 602 and the processing module 604 correspond to steps S502 to S506 in Example 3, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0135] Based on the solution provided by the above embodiment of the present application, after obtaining the neural network model that needs to be compressed, the neural network model can be trained using the singular value decomposition algorithm and the norm regularization algorithm to obtain a pre-trained model, and the weight matrix of the pre-trained model can be further pruned using the set target mask matrix, so that the acquired speech signal can be processed using the target model to obtain a recognition result, thereby achieving the purpose of data processing. Compared with the prior art, on the basis of SVD decomposition, the weight matrix is ​​continuously sparsely processed using the norm regularization algorithm, so that the sparsity of the weight matrix after SVD decomposition can be eliminated to the greatest extent, thereby achieving the technical effect of maximizing the reduction of the computational amount of the neural network model, further reducing the hardware cost and power consumption, and thus solving the technical problem of the large computational amount of the neural network model in the related technology, increasing the hardware cost and power consumption.

[0136] In the above embodiments of the present application, the processing module includes: a modification unit, a generation unit and a training unit.

[0137] Among them, the modification unit is used to modify the neural network model using the singular value decomposition algorithm to obtain a modified network model; the generation unit is used to generate a loss function based on the norm regularization algorithm and the target coefficient; the training unit is used to train the modified network model using the loss function and the target coefficient to obtain a pre-trained model.

[0138] In the above embodiments of the present application, the device further includes: a first training module and a determination module.

[0139] Among them, the acquisition module is also used to obtain multiple groups of initial coefficients; the first training module is used to train the modified network model using each group of initial coefficients, and determine the first precision and weight matrix corresponding to each group of initial coefficients; the acquisition module is also used to obtain the sparsity of the weight matrix corresponding to each group of initial coefficients, and obtain the matrix sparsity corresponding to each group of initial coefficients; the determination module is used to determine the target coefficients among the multiple groups of initial coefficients based on the first precision and matrix sparsity corresponding to each group of initial coefficients.

[0140] In the above embodiments of the present application, the determination module includes: a comparison unit and a first determination unit.

[0141] Among them, the comparison unit is used to compare the first precision corresponding to each group of initial coefficients with the original precision, and to compare the matrix sparsity corresponding to each group of initial coefficients with the preset sparsity; the first determination unit is used to determine that if the first precision is within the preset range of the original precision and the matrix sparsity is less than the preset sparsity, the corresponding initial coefficient is the target coefficient.

[0142] In the above embodiment of the present application, the device also includes: a first processing module, a second processing module, a second training module, a judgment module and a return module.

[0143] Among them, the first processing module is used to process the pre-trained model using the norm regularization algorithm to determine the mask matrix; the second processing module is used to process the weight matrix of the pre-trained model using the mask matrix to obtain the processed network model; the second training module is used to retrain the processed network model using the loss function and the target coefficient to obtain the trained network model; the judgment module is used to judge whether the second precision of the trained network model meets the target precision; the return module is used to use the trained network model as the pre-trained model if the second precision does not meet the target precision, and return to execute the function of the second processing module until the second precision meets the target precision; the determination module is used to determine that the mask matrix is ​​the target mask matrix if the second precision meets the target precision.

[0144] In the above embodiments of the present application, the first processing module includes: a processing unit, an acquisition unit, and a second determination unit.

[0145] Among them, the processing unit is used to perform norm processing on the weight matrix of the pre-trained model according to the grouping combination and column dimension to obtain the norm values ​​of multiple weight columns; the acquisition unit is used to obtain the weight column whose norm value is less than the first preset value to obtain the target weight column; the second determination unit is used to determine that the element value corresponding to the target weight column in the mask matrix is ​​the preset value.

[0146] In the above embodiments of the present application, the processing unit is also used to obtain the mean of the absolute values ​​of the weight values ​​located in the same column in the weight matrix according to the column dimension, and obtain the first mean of multiple weight columns; the acquisition unit is also used to obtain the mean of all weight values ​​in the weight matrix, and obtain the second mean; the acquisition unit is also used to obtain the ratio of the first mean to the second mean of multiple weight columns, and obtain the ratio of multiple weight columns; the acquisition unit is also used to obtain a weight column whose ratio is less than a second preset value, and obtain a target weight column; the second determination unit is also used to determine that the element value corresponding to the target weight column in the mask matrix is ​​a preset value.

[0147] In the above embodiment of the present application, the second training module is also used to process the processed network model using a gradient truncation algorithm.

[0148] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0149] Example 5

[0150] According to an embodiment of the present application, a processing system for a neural network model is also provided, including:

[0151] Processor; and

[0152] The memory is connected to the processor and is used to provide the processor with instructions for processing the following processing steps: training the neural network model using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model; obtaining a target mask matrix corresponding to the pre-trained model, wherein the dimension of the mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model; and processing the weight matrix of the pre-trained model using the mask matrix to obtain a target model.

[0153] Based on the solution provided by the above-mentioned embodiment of the present application, after obtaining the neural network model that needs to be compressed, the neural network model can be trained using the singular value decomposition algorithm and the norm regularization algorithm to obtain a pre-trained model, and the weight matrix of the pre-trained model can be further pruned using the set target mask matrix to achieve the purpose of compressing the neural network model to the maximum extent. Compared with the prior art, on the basis of SVD decomposition, the weight matrix is ​​continuously sparsely processed using the norm regularization algorithm, so that the sparsity of the weight matrix after SVD decomposition can be eliminated to the maximum extent, achieving the technical effect of maximizing the reduction of the computational amount of the neural network model, and further reducing the hardware cost and power consumption, thereby solving the technical problem of large computational amount of the neural network model in the related art, increasing hardware cost and power consumption.

[0154] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0155] Example 6

[0156] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0157] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0158] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the processing method of the neural network model: train the neural network model using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model; obtain a target mask matrix corresponding to the pre-trained model, wherein the dimension of the target mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model; and process the weight matrix of the pre-trained model using the target mask matrix to obtain a target model.

[0159] Optionally, the computer terminal may be configured as follows: Figure 1 The hardware structure shown in Figure 1 As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 102 and a memory 104 .

[0160] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the processing method and device of the neural network model in the embodiment of the present application, and the program instructions / modules corresponding to the data processing method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the processing method and data processing method of the neural network model mentioned above. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0161] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: train the neural network model using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model; obtain a target mask matrix corresponding to the pre-trained model, wherein the dimension of the target mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model; and process the weight matrix of the pre-trained model using the target mask matrix to obtain a target model.

[0162] Optionally, the processor may also execute the program code of the following steps: modifying the neural network model using a singular value decomposition algorithm to obtain a modified network model; generating a loss function based on a norm regularization algorithm and a target coefficient; and training the modified network model using the loss function and the target coefficient to obtain a pre-trained model.

[0163] Optionally, the processor may also execute the program code of the following steps: obtaining multiple sets of initial coefficients before generating a loss function based on a norm regularization algorithm and a target coefficient; training the modified network model using each set of initial coefficients to determine a first precision and a weight matrix corresponding to each set of initial coefficients; obtaining the sparsity of the weight matrix corresponding to each set of initial coefficients to obtain the matrix sparsity corresponding to each set of initial coefficients; and determining the target coefficients among the multiple sets of initial coefficients based on the first precision and the matrix sparsity corresponding to each set of initial coefficients.

[0164] Optionally, the processor may also execute the following program code: comparing the first precision corresponding to each group of initial coefficients with the original precision, and comparing the matrix sparsity corresponding to each group of initial coefficients with the preset sparsity; if the first precision is within the preset range of the original precision and the matrix sparsity is less than the preset sparsity, then determining the corresponding initial coefficient as the target coefficient.

[0165] Optionally, the processor may also execute the following program code: before obtaining the target mask matrix corresponding to the pre-trained model, use the norm regularization algorithm to process the pre-trained model to determine the mask matrix; use the mask matrix to process the weight matrix of the pre-trained model to obtain the processed network model; use the loss function and the target coefficient to retrain the processed network model to obtain the trained network model; determine whether the second precision of the trained network model meets the target precision; if the second precision does not meet the target precision, use the trained network model as the pre-trained model, and return to execute the step of processing the pre-trained model using the norm regularization algorithm to determine the mask matrix until the second precision meets the target precision; if the second precision meets the target precision, determine that the mask matrix is ​​the target mask matrix.

[0166] Optionally, the processor may also execute the program code of the following steps: performing norm processing on the weight matrix of the pre-trained model according to the grouping combination and column dimension to obtain the norm values ​​of multiple weight columns; obtaining the weight column whose norm value is less than the first preset value to obtain the target weight column; and determining that the element value corresponding to the target weight column in the mask matrix is ​​the preset value.

[0167] Optionally, the processor may also execute the program code of the following steps: obtaining the mean of the absolute values ​​of the weight values ​​in the same column of the weight matrix according to the column dimension to obtain a first mean of multiple weight columns; obtaining the mean of all weight values ​​in the weight matrix to obtain a second mean; obtaining the ratio of the first mean to the second mean of multiple weight columns to obtain a ratio of multiple weight columns; obtaining a weight column whose ratio is less than a second preset value to obtain a target weight column; and determining that the element value corresponding to the target weight column in the mask matrix is ​​a preset value.

[0168] Optionally, the processor may also execute the program code of the following steps: in the process of retraining the processed network model using the loss function and the target coefficient, the processed network model is processed using a gradient truncation algorithm.

[0169] Optionally, the processor may also execute program codes of the following steps: obtaining a speech signal to be processed; and recognizing the speech signal using a target model to obtain a recognition result.

[0170] By adopting the embodiment of the present application, a processing scheme for a neural network model is provided. Based on SVD decomposition, the weight matrix is ​​continuously sparsed using a norm regularization algorithm, so that the sparsity of the weight matrix after SVD decomposition can be eliminated to the greatest extent, thereby achieving the technical effect of maximizing the reduction of the computational amount of the neural network model, further reducing the hardware cost and power consumption, and thus solving the technical problem of large computational amount of the neural network model in related technologies, increasing hardware cost and power consumption.

[0171] It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 1 The structure of the electronic device is not limited. For example, the computer terminal 10 may also include Figure 1 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 1 Different configurations are shown.

[0172] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0173] Example 7

[0174] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the processing method of the neural network model provided in the first embodiment.

[0175] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0176] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: training the neural network model using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model; obtaining a target mask matrix corresponding to the pre-trained model, wherein the dimension of the target mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model; and processing the weight matrix of the pre-trained model using the target mask matrix to obtain a target model.

[0177] Optionally, the storage medium is also configured to store program codes for executing the following steps: modifying the neural network model using a singular value decomposition algorithm to obtain a modified network model; generating a loss function based on a norm regularization algorithm and a target coefficient; and training the modified network model using the loss function and the target coefficient to obtain a pre-trained model.

[0178] Optionally, the storage medium is also configured to store program code for executing the following steps: before generating a loss function based on a norm regularization algorithm and a target coefficient, obtaining multiple sets of initial coefficients; training the modified network model using each set of initial coefficients to determine a first precision and a weight matrix corresponding to each set of initial coefficients; obtaining the sparsity of the weight matrix corresponding to each set of initial coefficients to obtain a matrix sparsity corresponding to each set of initial coefficients; and determining a target coefficient among the multiple sets of initial coefficients based on the first precision and the matrix sparsity corresponding to each set of initial coefficients.

[0179] Optionally, the storage medium is also configured to store program code for executing the following steps: comparing the first precision corresponding to each group of initial coefficients with the original precision, and comparing the matrix sparsity corresponding to each group of initial coefficients with the preset sparsity; if the first precision is within a preset range of the original precision and the matrix sparsity is less than the preset sparsity, then determining the corresponding initial coefficient as the target coefficient.

[0180] Optionally, the storage medium is also configured to store program codes for executing the following steps: before obtaining the target mask matrix corresponding to the pre-trained model, the pre-trained model is processed using a norm regularization algorithm to determine the mask matrix; the weight matrix of the pre-trained model is processed using the mask matrix to obtain a processed network model; the processed network model is retrained using a loss function and a target coefficient to obtain a trained network model; it is determined whether the second precision of the trained network model meets the target precision; if the second precision does not meet the target precision, the trained network model is used as a pre-trained model, and the step of processing the pre-trained model using a norm regularization algorithm to determine the mask matrix is ​​returned until the second precision meets the target precision; if the second precision meets the target precision, the mask matrix is ​​determined to be the target mask matrix.

[0181] Optionally, the storage medium is also configured to store program codes for executing the following steps: performing norm processing on the weight matrix of the pre-trained model according to the grouping combination and column dimension to obtain norm values ​​of multiple weight columns; obtaining a weight column whose norm value is less than a first preset value to obtain a target weight column; and determining that the element value corresponding to the target weight column in the mask matrix is ​​a preset value.

[0182] Optionally, the storage medium is also configured to store program codes for executing the following steps: obtaining the mean of the absolute values ​​of the weight values ​​located in the same column in the weight matrix according to the column dimension to obtain a first mean of multiple weight columns; obtaining the mean of all weight values ​​in the weight matrix to obtain a second mean; obtaining the ratio of the first mean to the second mean of multiple weight columns to obtain a ratio of multiple weight columns; obtaining a weight column whose ratio is less than a second preset value to obtain a target weight column; and determining that the element value corresponding to the target weight column in the mask matrix is ​​a preset value.

[0183] Optionally, the storage medium is also configured to store program codes for executing the following steps: in the process of retraining the processed network model using the loss function and the target coefficient, the processed network model is processed using a gradient truncation algorithm.

[0184] Optionally, the storage medium is further configured to store program codes for executing the following steps: obtaining a speech signal to be processed; and recognizing the speech signal using a target model to obtain a recognition result.

[0185] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0186] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0187] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0188] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0190] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0191] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing a neural network model, comprising: The neural network model is trained using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model, wherein the pre-trained model is a model obtained by performing structured sparseness on a weight matrix of the neural network model using a norm regularization algorithm after performing singular value decomposition on the neural network model, and the pre-trained model is trained using a loss function and a target coefficient, the loss function is determined based on the norm regularization algorithm and the target coefficient, and the target coefficient is determined based on a comparison result between a first precision corresponding to each group of initial coefficients and an original precision, and a comparison result between a matrix sparsity and a preset sparsity; Obtain a target mask matrix corresponding to the pre-trained model, wherein the dimension of the target mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model; The weight matrix of the pre-trained model is processed using the target mask matrix to obtain a target model, wherein the target model is used to recognize speech signals.

2. The method according to claim 1, wherein: The neural network model is trained using the singular value decomposition algorithm and the norm regularization algorithm, and the pre-trained model includes: Using a singular value decomposition algorithm to modify the neural network model to obtain a modified network model; Generate a loss function based on the norm regularization algorithm and the target coefficient; The modified network model is trained using the loss function and the target coefficient to obtain the pre-trained model.

3. The method according to claim 2, wherein: Before generating the loss function based on the norm regularization algorithm and the target coefficient, the method further includes: Get multiple sets of initial coefficients; Using each set of initial coefficients to train the modified network model, and determining a first precision and a weight matrix corresponding to each set of initial coefficients; Obtaining the sparsity of the weight matrix corresponding to each group of initial coefficients to obtain the sparsity of the matrix corresponding to each group of initial coefficients; The target coefficients in the multiple groups of initial coefficients are determined based on the first precision and matrix sparsity corresponding to each group of initial coefficients.

4. The method according to claim 3, wherein: Determining the target coefficients in the multiple groups of initial coefficients based on the first precision and the matrix sparsity corresponding to each group of initial coefficients includes: Compare the first precision corresponding to each group of initial coefficients with the original precision, and compare the matrix sparsity corresponding to each group of initial coefficients with the preset sparsity; If the first precision is within a preset range of the original precision and the matrix sparsity is less than the preset sparsity, the corresponding initial coefficient is determined to be the target coefficient.

5. The method according to claim 2, wherein: Before obtaining the target mask matrix corresponding to the pre-trained model, the method further includes: Processing the pre-trained model using a norm regularization algorithm to determine a mask matrix; Using the mask matrix to process the weight matrix of the pre-trained model to obtain a processed network model; Retraining the processed network model using the loss function and the target coefficient to obtain a trained network model; Determining whether the second accuracy of the trained network model meets the target accuracy; If the second precision does not meet the target precision, the trained network model is used as the pre-trained model, and the step of processing the pre-trained model using the norm regularization algorithm to determine the mask matrix is ​​returned to execute until the second precision meets the target precision; If the second precision meets the target precision, the mask matrix is ​​determined to be the target mask matrix.

6. The method according to claim 5, wherein: The pre-trained model is processed using a norm regularization algorithm to determine a mask matrix including: Performing norm processing on the weight matrix of the pre-trained model according to the grouping combination and the column dimension to obtain norm values ​​of multiple weight columns; Obtain a weight column whose norm value is less than a first preset value to obtain a target weight column; Determine that the element value corresponding to the target weight column in the mask matrix is ​​a preset value.

7. The method according to claim 5, wherein: The pre-trained model is processed using a norm regularization algorithm to determine a mask matrix including: Obtaining the mean of the absolute values ​​of the weight values ​​in the same column in the weight matrix according to the column dimension, to obtain a first mean value of the plurality of weight columns; Obtaining the mean of all weight values ​​in the weight matrix to obtain a second mean; Obtaining a ratio of a first mean value of the plurality of weight columns to the second mean value to obtain a ratio of the plurality of weight columns; Obtain a weight column whose ratio is less than a second preset value to obtain a target weight column; Determine that the element value corresponding to the target weight column in the mask matrix is ​​a preset value.

8. The method according to claim 5, wherein: In the process of retraining the processed network model using the loss function and the target coefficient, the method further includes: The processed network model is processed using a gradient truncation algorithm.

9. The method according to claim 1, wherein: After the weight matrix of the pre-trained model is processed by using the target mask matrix to obtain a target model, the method further includes: Obtaining a speech signal to be processed; The target model is used to recognize the speech signal to obtain a recognition result.

10. A data processing method, comprising: Obtaining a speech signal to be processed; The speech signal is processed using a target model to obtain a recognition result, wherein the target model is obtained by processing a weight matrix of a pre-trained model using a target mask matrix, the pre-trained model is obtained by training a neural network model using a singular value decomposition algorithm and a norm regularization algorithm, the dimension of the target mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model, the pre-trained model is a model obtained by performing structured sparseness on the weight matrix of the neural network model using a norm regularization algorithm after performing singular value decomposition on the neural network model, the target model is used to recognize speech signals, the pre-trained model is trained using a loss function and a target coefficient, the loss function is determined based on the norm regularization algorithm and the target coefficient, and the target coefficient is determined based on a comparison result between a first precision corresponding to each group of initial coefficients and an original precision, and a comparison result between a matrix sparsity and a preset sparsity.

11. The method according to claim 10, wherein: Training the neural network model using the singular value decomposition algorithm and norm regularization algorithm includes: Using a singular value decomposition algorithm to modify the neural network model to obtain a modified network model; Generate a loss function based on the norm regularization algorithm and the target coefficient; The modified network model is trained using the loss function and the target coefficient.

12. A processing device for a neural network model, comprising: A first training module is used to train the neural network model using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model, wherein the pre-trained model is a model obtained by performing structured sparseness on a weight matrix of the neural network model using a norm regularization algorithm after performing singular value decomposition on the neural network model, and the pre-trained model is trained using a loss function and a target coefficient, the loss function is determined based on the norm regularization algorithm and the target coefficient, and the target coefficient is determined based on a comparison result between a first precision corresponding to each group of initial coefficients and an original precision, and a comparison result between a matrix sparsity and a preset sparsity; An acquisition module, used to acquire a target mask matrix corresponding to the pre-trained model, wherein the dimension of the mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model; The first processing module is used to use the mask matrix to process the weight matrix of the pre-trained model to obtain a target model, wherein the target model is used to recognize the speech signal.

13. The device according to claim 12, wherein: The first training module includes: A modification unit, used to modify the neural network model using a singular value decomposition algorithm to obtain a modified network model; A generation unit, used to generate a loss function based on a norm regularization algorithm and a target coefficient; A training unit is used to train the modified network model using the loss function and the target coefficient to obtain the pre-trained model.

14. The device according to claim 13, wherein: The device also includes: A second processing module is used to process the pre-trained model using a norm regularization algorithm to determine a mask matrix; A third processing module, used to process the weight matrix of the pre-trained model using the mask matrix to obtain a processed network model; A second training module is used to retrain the processed network model using the loss function and the target coefficient to obtain a trained network model; A judgment module, used to judge whether the second accuracy of the trained network model meets the target accuracy; A return module, configured to use the trained network model as the pre-trained model if the second accuracy does not meet the target accuracy, and return to execute the function of the second processing module until the second accuracy meets the target accuracy; A determination module is configured to determine that the mask matrix is ​​the target mask matrix if the second precision satisfies the target precision.

15. A data processing device, comprising: An acquisition module, used for acquiring a speech signal to be processed; A processing module is used to recognize the speech signal using a target model to obtain a recognition result, wherein the target model is obtained by processing the weight matrix of a pre-trained model using a target mask matrix, the pre-trained model is obtained by training a neural network model using a singular value decomposition algorithm and a norm regularization algorithm, the dimension of the target mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model, the pre-trained model is a model obtained by performing structured sparseness on the weight matrix of the neural network model using a norm regularization algorithm after performing singular value decomposition on the neural network model, the target model is used to recognize the speech signal, the pre-trained model is trained using a loss function and a target coefficient, the loss function is determined based on the norm regularization algorithm and the target coefficient, and the target coefficient is determined based on the comparison result of the first precision corresponding to each group of initial coefficients with the original precision, and the comparison result of the matrix sparsity with the preset sparsity.

16. The device according to claim 15, wherein: The processing module comprises: A modification unit, used to modify the neural network model using a singular value decomposition algorithm to obtain a modified network model; A generation unit, used to generate a loss function based on a norm regularization algorithm and a target coefficient; A training unit is used to train the modified network model using the loss function and the target coefficient.

17. A storage medium comprising a stored program, wherein: When the program is running, the device where the storage medium is located is controlled to execute the processing method of the neural network model described in any one of claims 1 to 8, or the data processing method described in any one of claims 10 to 11.

18. A computing device comprising: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein when the program is run, the processing method of the neural network model described in any one of claims 1 to 8, or the data processing method described in any one of claims 10 to 11 is executed.

19. A processing system for a neural network model, comprising: processor; as well as A memory connected to the processor is used to provide the processor with instructions for processing the following processing steps: training a neural network model using a singular value decomposition algorithm and a norm regularization algorithm to obtain a pre-trained model; obtaining a target mask matrix corresponding to the pre-trained model, wherein the dimension of the mask matrix is ​​the same as the dimension of the weight matrix of the pre-trained model; processing the weight matrix of the pre-trained model using the mask matrix to obtain a target model, wherein the pre-trained model is a model obtained by performing structured sparseness on the weight matrix of the neural network model using a norm regularization algorithm after performing singular value decomposition on the neural network model, and wherein the target model is used to recognize speech signals, wherein the pre-trained model is trained using a loss function and a target coefficient, wherein the loss function is determined based on the norm regularization algorithm and the target coefficient, and wherein the target coefficient is determined based on a comparison result between a first precision corresponding to each group of initial coefficients and an original precision, and a comparison result between a matrix sparsity and a preset sparsity.

Images