Model sparse method and device, electronic equipment and storage medium

Abstract

The invention provides a model sparse method and device, electronic equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: adding auxiliary parameters into an original model by the electronic equipment to obtain an intermediate model; the auxiliary parameters are used for adjusting the precision of the intermediate model. And the electronic equipment adjusts the auxiliary parameters in the sparse process of the intermediate model to obtain a target model. The original model, the intermediate model and the target model can be collectively referred to as neural network models. According to the embodiment of the invention, in the process of sparse processing of the neural network model, the precision of the neural network model can be improved by adjusting the auxiliary parameters, and the loss of model performance caused by sparse processing of the model is reduced; and the auxiliary parameters are adjusted, and all parameters of the sparse neural network model do not need to be adjusted, so that the number of the parameters needing to be adjusted is reduced, computing resources consumed in the model rarefaction process can be reduced, the model rarefaction speed can be increased, and the efficiency of the model rarefaction process can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to a model sparsity method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of artificial intelligence (AI) technology, large-scale neural network models are being used more and more widely in people's daily work and life. Inference using large-scale neural network models typically consumes a large amount of computing resources and requires substantial storage space to store model parameters. For resource-constrained devices, such as mobile devices, edge computing nodes, and low-power embedded devices, using neural network models for inference presents a significant challenge.

[0003] To reduce the resource consumption of neural network models, sparsification can be applied to remove a large number of redundant model parameters, thus simplifying the model. However, sparsified neural network models typically suffer significant losses in accuracy and performance. Summary of the Invention

[0004] This application provides a model sparsity method, apparatus, electronic device, and storage medium, which can not only improve the accuracy and performance of the neural network model after sparsification, but also improve the efficiency of the model sparsification process.

[0005] Firstly, this application provides a model sparsity method, which can be executed by an electronic device, or a chip, chip system, or circuit within the electronic device. For example, the electronic device can be a computer, server, tablet computer, laptop computer, vehicle terminal, etc. The method can include: the electronic device adding auxiliary parameters to the original model to obtain an intermediate model; wherein the auxiliary parameters are used to adjust the accuracy of the intermediate model. During the sparsification process of the intermediate model, the electronic device adjusts the auxiliary parameters to obtain the target model.

[0006] In this application, the original model, intermediate model, and target model can be collectively referred to as a neural network model. In the process of sparsifying the neural network model, the embodiments of this application can improve the accuracy of the neural network model and reduce the performance loss caused by model sparsification by adjusting auxiliary parameters. Furthermore, by adjusting the auxiliary parameters to regulate the accuracy of the neural network model, it is not necessary to adjust all parameters of the sparsified neural network model, reducing the number of parameters that need to be adjusted. This reduces the computational resources consumed in the model sparsification process, accelerates the model sparsification speed, and improves the efficiency of the model sparsification process.

[0007] In one possible implementation, the electronic device can determine the model sparsity rate for each sparsity processing based on the set target sparsity and the set number of sparsity iterations N, and perform N sparsity processing on the intermediate model according to the model sparsity rate for each sparsity processing; the auxiliary parameters are adjusted for each sparsity processing.

[0008] In the above implementation, by sparsifying the neural network model multiple times instead of once, the sudden drop in model accuracy caused by parameter mutations due to a single sparsity operation can be reduced.

[0009] In one possible implementation, the intermediate model comprises multiple network layers. During each sparsity processing step, the electronic device can determine the layer sparsity rate of each network layer based on the model sparsity rate of this sparsity processing and the weight matrix of each network layer. According to the layer sparsity rate of each network layer, some parameters in each network layer are removed to obtain the sparsified intermediate model. Then, the auxiliary parameters in the sparsified intermediate model are adjusted using a validation dataset.

[0010] In the above implementation, the sparsity rate of each network layer can be determined based on the weight matrix of the network layers. More important network layers have lower sparsity rates, while less important network layers have higher sparsity rates. This allows for setting different sparsity rates for different network layers based on their importance, enabling targeted application of different sparsity rates to different network layers. This helps maintain the accuracy of the large model after sparsification and reduces accuracy loss. After obtaining the sparse intermediate model, adjusting the auxiliary parameters in the sparse intermediate model can further improve the accuracy of the large model after sparsification.

[0011] In one possible implementation, when the electronic device adjusts the auxiliary parameters in the sparse intermediate model using a validation dataset, it can use the sparse intermediate model as the model to be updated and repeatedly execute the following steps: inputting the data in the validation dataset into the model to be updated to obtain the prediction result output by the model to be updated; comparing the prediction result with the reference result to obtain the loss value; adjusting the auxiliary parameters in the model to be updated according to the obtained loss value to obtain a new model to be updated, and returning to execute the step of inputting the data in the validation dataset into the model to be updated until the obtained loss value converges.

[0012] By adjusting the auxiliary parameters in the model through the above iterative process until the loss value converges, the accuracy of the sparsed model can be guaranteed.

[0013] In one possible implementation, the sparsity of the model gradually decreases from the first sparsification to the Nth sparsification.

[0014] In the above implementation, by controlling the model sparsity rate of each iteration step, the neural network model is progressively sparsified according to the model sparsity rate from large to small. This can reduce the sudden drop in model accuracy caused by parameter mutations. Throughout the process, redundant parameters are sparsified as much as possible, and the sparsity is increased from small to large. The model accuracy is restored at the layer level. The goal is to reduce the loss of model accuracy while ensuring the target sparsity.

[0015] In one possible implementation, the auxiliary parameters include a layer adaptation parameter added before each network layer of the original model and a scaling bias parameter added to each network layer of the original model.

[0016] In the above implementation, by adjusting only the hierarchical adaptive parameters and scaling bias parameters, the number of parameters that need to be adjusted is reduced, which can reduce the computational resources consumed in the model sparsification process and ensure the accuracy of the large model after sparsification at low cost.

[0017] In one possible implementation, the original model is a large, pre-trained language model.

[0018] Secondly, this application provides a model sparse device that can be applied in electronic devices, and the model sparse device may include:

[0019] The parameter addition module is used to add auxiliary parameters to the original model to obtain an intermediate model; the auxiliary parameters are used to adjust the accuracy of the intermediate model.

[0020] The parameter adjustment module is used to adjust auxiliary parameters during the sparsification process of the intermediate model to obtain the target model.

[0021] In one possible implementation, the parameter adjustment module is specifically used for:

[0022] Based on the set target sparsity and the set number of sparsity iterations N, the model sparsity rate for each sparsity processing is determined.

[0023] Based on the sparsity rate of the model in each sparsity process, the intermediate model is subjected to N sparsity processes; the auxiliary parameters are adjusted in each sparsity process.

[0024] In one possible implementation, the intermediate model includes multiple network layers; the parameter tuning module is specifically used for:

[0025] In each sparsity processing process, the layer sparsity of each network layer is determined based on the model sparsity of this sparsity processing and the weight matrix of each network layer in multiple network layers.

[0026] Based on the sparsity of each network layer, some parameters in each network layer are removed to obtain a sparse intermediate model.

[0027] The auxiliary parameters in the sparsed intermediate model are adjusted using a validation dataset.

[0028] In one possible implementation, the parameter adjustment module is specifically used for:

[0029] Use the sparsed intermediate model as the model to be updated, and repeat the following steps:

[0030] Input the data from the validation dataset into the model to be updated, and obtain the prediction results output by the model to be updated;

[0031] The predicted results are compared with the reference results to obtain the loss value;

[0032] Adjust the auxiliary parameters in the model to be updated based on the obtained loss value to obtain a new model to be updated, and return to the step of inputting the data in the validation dataset into the model to be updated until the obtained loss value converges.

[0033] In one possible implementation, the sparsity of the model gradually decreases from the first sparsification to the Nth sparsification.

[0034] In one possible implementation, the auxiliary parameters include a layer adaptation parameter added before each network layer of the original model and a scaling bias parameter added to each network layer of the original model.

[0035] In one possible implementation, the original model is a large, pre-trained language model.

[0036] Thirdly, this application also provides an electronic device including at least one processor and at least one memory. The one or more memories store one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform any of the methods described in the first aspect above.

[0037] Fourthly, this application also provides a chip including a processor and a power supply circuit. The power supply circuit is used to supply power to the processor, which is used to execute a computer program to implement any of the methods described in the first aspect above.

[0038] Fifthly, this application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform any of the methods provided in the first aspect above.

[0039] In a sixth aspect, this application provides a computer program product storing instructions that, when executed by a processor, implement any of the methods provided in the first aspect.

[0040] The technical effects that can be achieved by any of the technical solutions in the second to sixth aspects mentioned above can be described with reference to the technical effects that can be achieved by the technical solution in the first aspect mentioned above, and the repeated parts will not be repeated. Attached Figure Description

[0041] Figure 1 A flowchart of a model sparsity method provided in an embodiment of this application;

[0042] Figure 2 A schematic diagram of a model with added auxiliary parameters provided in an embodiment of this application;

[0043] Figure 3 A schematic diagram illustrating the sparsity of a model with different iteration numbers, provided for an embodiment of this application;

[0044] Figure 4 A schematic diagram illustrating the layer sparsity of different network layers provided in this application embodiment;

[0045] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0046] Figure 6 This is a schematic diagram of the structure of a model sparse device provided in an embodiment of this application;

[0047] Figure 7 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the embodiments of this application will be described in detail below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application.

[0049] Before introducing the specific solutions provided in the embodiments of this application, some terms used in this application will be explained to facilitate understanding by those skilled in the art, but the terms used in this application are not limited.

[0050] (1) Large Language Model (LLM): An LLM is an extremely large-scale language model, consisting of artificial neural networks with billions of weights or more parameters. It uses unsupervised learning to pre-train on large amounts of unlabeled corpora to generate a general LLM model, which can then be fine-tuned for different specific tasks. For example, LLMs can be used for generative tasks such as text generation, machine translation, and dialogue generation. LLMs are typically based on the Transformer architecture; in other words, LLMs can accomplish generative tasks by stacking multiple Transformer layers.

[0051] (2) Model sparsification: Model sparsification refers to reducing the number of parameters in a large model by eliminating some of the parameters in the model, thereby reducing the resource consumption of the large model. A large model refers to a large-scale neural network model containing billions or hundreds of billions of parameters, such as LLM.

[0052] In this application embodiment, "multiple" refers to two or more. Therefore, in this application embodiment, "multiple" can also be understood as "at least two". "At least one" can be understood as one or more, such as one, two, or more. For example, "including at least one" means including one, two, or more, and it does not limit which ones are included. For example, including at least one of A, B, and C, then it could include A, B, C, A and B, A and C, B and C, or A and B and C. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0053] Unless otherwise stated, the ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used to distinguish multiple objects, and are not used to limit the order, sequence, priority or importance of multiple objects.

[0054] With the development of AI technology, neural network models are being used more and more widely in people's daily work and life. Furthermore, the size of neural network models is constantly increasing, as are their resource consumption. The training and inference processes of neural network models typically require significant computational resources and substantial storage space to store model parameters. For resource-constrained devices, such as mobile devices, edge computing nodes, and low-power embedded devices, using neural network models for inference presents a huge challenge. Moreover, neural network models containing hundreds of billions of parameters are limited by computing power, double data rate (DDR) memory capacity, and memory access bandwidth, making it difficult to achieve low inference latency or high throughput. This is often the main bottleneck limiting the application scope of large models.

[0055] To enable the wider application of neural network models across various devices, and to improve the accuracy and performance of the sparsed neural network model while effectively reducing the computational resources required for its operation and enhancing its inference efficiency, this application provides a model sparsification method. This model sparsification method can be executed by an electronic device, which may include, but is not limited to, servers, computers, tablets, vehicle terminals, smart medical terminals, smart grid terminals, and mobile terminals. The model sparsification method may include: adding auxiliary parameters to the original model to obtain an intermediate model, wherein the auxiliary parameters are used to adjust the accuracy of the intermediate model; and adjusting the auxiliary parameters during the sparsification process of the intermediate model to obtain the target model. The original model, intermediate model, and target model described above can be collectively referred to as a neural network model. In this embodiment of the application, during the sparsification process of the neural network model, the accuracy of the neural network model can be improved by adjusting auxiliary parameters, reducing the loss of model performance caused by model sparsification. Furthermore, by adjusting the auxiliary parameters to adjust the accuracy of the neural network model, it is not necessary to adjust all parameters of the sparsified neural network model, reducing the number of parameters that need to be adjusted. This reduces the computational resources consumed in the model sparsification process, speeds up the model sparsification process, and improves the efficiency of the model sparsification process.

[0056] Figure 1 An exemplary flowchart of a model sparsity method provided in an embodiment of this application is shown. Figure 1 As shown, the method may include the following steps:

[0057] S101, add auxiliary parameters to the original model to obtain the intermediate model.

[0058] The electronic device acquires the original model, which can be a trained LLM, another large model, or another neural network model. Auxiliary parameters are added to the original model to adjust the accuracy of the intermediate model. These auxiliary parameters may include layer adaptation parameters added before each network layer of the original model and scaling & bias parameters added to each network layer of the original model.

[0059] For example, such as Figure 2 As shown, the original model can include a multi-layer decoder structure, and each decoder layer can be a Transformer structure, called a network layer, such as... Figure 2 The diagram shows layers 1, 2, ..., L. Layer adaptive parameters can be added before each layer. These parameters act externally to each network layer. For example, they can be added to the attention module before each layer, serving as a prefix to the key-value pairs input to each layer during training or inference. Layer adaptive parameters can guide the sparsed neural network model in generating contextual responses, reducing reconstruction errors.

[0060] Scaling and bias parameters can be added to each network layer. For example, scaling and bias parameters can be added to each layer of a feedforward neural network (FFN) structure. These scaling and bias parameters can include a scaling parameter and a bias parameter. For instance, an FFN structure can include three linear layers, each of which can have both scaling and bias parameters added. Each network layer can also include an attention structure, and each attention structure can include four linear layers; similarly, each linear layer can have both scaling and bias parameters added. Furthermore, each linear layer includes a weight matrix, and the scaling and bias parameters can be added to the weight matrix row-by-row. Assuming the weight matrix has five rows, the scaling parameter can be a 5x1 column vector, and the bias parameter can also be a 5x1 column vector. One element from the scaling parameter and one element from the bias parameter can be added to each row of the weight matrix. The scaling and bias parameters are used to scale and fine-tune the bias of each row of the weight matrix to reduce the feature error offset of the linear layer output in the attention structure and FFN structure, and further reduce the reconstruction error of the sparse neural network model.

[0061] S102, based on the set target sparsity and the set number of sparsity iterations n, determine the model sparsity rate for each sparsity processing step.

[0062] Considering that a single-shot sparsity operation removing a large number of weights from a neural network model can significantly impact its performance, fine-tuning may not fully compensate for the information loss caused by the sudden removal of weights. Therefore, this embodiment of the application performs sparsity processing on the neural network model multiple times, removing a small portion of the weights in each sparsity process, and then fine-tunes the parameters to restore the performance of the neural network model, thereby reducing the performance loss of the neural network model.

[0063] In this embodiment, the electronic device includes a progressive sparsity module. The electronic device receives a target sparsity input by the user and, through the progressive sparsity module, determines the model sparsity rate for each sparsity processing step based on the target sparsity and a set number of sparsity iterations, n. In some embodiments, the model sparsity rate can be gradually reduced from the first sparsity processing step to the Nth sparsity processing step, performing progressive sparsity on the neural network model. For example, the model sparsity rate for each sparsity processing step can be determined by the following formula:

[0064]

[0065] Among them, s i Indicates the initial sparsity, which defaults to 0; s f The target sparsity is represented by t; the current iteration number or step number is represented by s. t This represents the sparsity rate of the model corresponding to the current step; n represents the set number of sparse iterations, i.e., the total number of sparse iterations.

[0066] For example, such as Figure 3As shown, assuming the target sparsity is 0.75 and the set number of sparse iterations n is 10, the model sparsity corresponding to each sparsification process decreases from large to small. The model sparsity corresponding to the first sparsification process is 0.203, the model sparsity corresponding to the second sparsification process is 0.366-0.203=0.163, the model sparsity corresponding to the third sparsification process is 0.493-0.366=0.127, the model sparsity corresponding to the fourth sparsification process is 0.588-0.493=0.095, and the model sparsity corresponding to the fifth sparsification process is 0.656- 0.588 = 0.068. The sparsity of the model corresponding to the sixth sparsity step is 0.702 - 0.656 = 0.046, the seventh sparsity step is 0.730 - 0.702 = 0.028, the eighth sparsity step is 0.744 - 0.730 = 0.014, the ninth sparsity step is 0.749 - 0.744 = 0.005, and the tenth sparsity step is 0.750 - 0.749 = 0.001, finally reaching the target sparsity of 0.75. Therefore, it can be seen that as the number of iterations increases, the sparsity gradually decreases. Figure 3 The slope of the curve represents the sparsity of the neural network model in each sparse processing step.

[0067] S103. Based on the model sparsity corresponding to the current iteration number and the weight matrix of each network layer, determine the layer sparsity of each network layer.

[0068] Since the redundancy levels differ across layers in a large model, this embodiment of the application can determine the sparsity rate of each network layer based on the outlier ratio. In this embodiment, the electronic device includes a non-uniform sparsity module for determining the sparsity rate of each network layer based on the outlier ratio. The outlier ratio is determined by the weight matrix of each network layer; more important network layers have lower sparsity rates, while less important network layers have higher sparsity rates. For example, the outlier ratio of any network layer can be determined using the following formula:

[0069]

[0070] Among them, A ij =∥X j ∥2·|W ij |, Let N be the input matrix of any network layer, N be the batch size of the input data, and L be the length of the input data sequence. Let be the weight matrix of any network layer, including added scaling and bias parameters. Let ij be the row * column of the weight matrix. `mean()` is the averaging function, and `I()` is the conditional function; if A... ij If the condition `mean(A)×W` is met, `I()` returns 1; otherwise, it returns 0. `M∈[3,5,7,10]` is a hyperparameter. It calculates the outlier ratio of a network layer by identifying the proportion of elements whose size is M times greater than the average value of elements A in any given network layer. A higher outlier ratio indicates a more important network layer, while a lower outlier ratio indicates a less important network layer.

[0071] Using the above formula, we can obtain the outlier ratio D = [D1, D2, ..., D] of all network layers in the neural network model. n The ratio of outliers in different network layers is inversely proportional to the sparsity of the corresponding network layers. Assuming the sparsity of all network layers in a neural network model is S = [S1, S2, ..., S...], ... n ], then we have S i ∝1-D i The value of i ranges from 1 to n. While adhering to the principle that network layers with a high outlier ratio should have a low layer sparsity, the sum of the layer sparsities of all network layers is kept equal to the model sparsity corresponding to the current iteration number.

[0072] For example, such as Figure 4 As shown, the horizontal axis represents the numbers of multiple different network layers, and the vertical axis represents the sparsity. Figure 4 The dashed line represents the average sparsity of multiple network layers. For example, if a neural network model has 32 network layers and the sparsity of the model corresponding to the current iteration number is 0.32, then the average sparsity of the 32 network layers is 0.32 / 32 = 0.01. Figure 4 The broken line in the figure represents the variation in the sparsity of different network layers. As can be seen from the broken line, the sparsity of each network layer is different.

[0073] S104. Based on the sparsity of each network layer, remove some parameters from each network layer of the intermediate model to obtain the sparse intermediate model.

[0074] For example, for the i-th network layer, the parameters with smaller absolute values ​​in the i-th network layer can be removed according to the layer sparsity rate of the i-th network layer. That is, the elements with smaller absolute values ​​in the weight matrix of the i-th network layer are set to 0. For instance, assuming the layer sparsity rate of the i-th network layer is 0.02, the elements in the weight matrix of the i-th network layer can be sorted in ascending order of absolute value, and the first 2% of elements are set to 0. After removing some parameters from each network layer, a sparse intermediate model is obtained.

[0075] S105, the auxiliary parameters in the sparsed intermediate model are adjusted using the validation dataset to obtain an updated intermediate model.

[0076] The validation dataset can be obtained in advance and may include reference results. Alternatively, the reference results can be obtained in advance by inputting the data in the validation dataset into the original model, obtaining the reference results output by the original model, and saving the obtained reference results.

[0077] In some embodiments, after obtaining the sparse intermediate model, it can be used as the model to be updated, and the following steps are repeated: inputting data from the validation dataset into the model to be updated, obtaining the prediction result output by the model to be updated, comparing the prediction result with the reference result to obtain the loss value, adjusting the auxiliary parameters in the model to be updated based on the obtained loss value, obtaining a new model to be updated, and returning to the step of inputting data from the validation dataset into the model to be updated until the obtained loss value converges, thus obtaining an updated intermediate model to recover the model accuracy error caused by weight sparsity. In the above process, only the layer adaptive parameters and scaling bias parameters added to the neural network model are updated; other parameters in the neural network model are frozen and not updated. Therefore, the number of parameters that need to be updated can be greatly reduced. The updated parameters account for only 0.06% of the total parameters of the neural network model, occupying approximately 8MB of storage space, significantly reducing the time and resource consumption of the entire model's sparse fine-tuning and improving efficiency. By adding layer adaptive parameters and scaling bias parameters to the neural network model, lightweight and efficient sparse error recovery can be achieved.

[0078] After each sparsity adjustment, appropriate fine-tuning can be performed to restore the performance of the neural network model, which can further improve the model accuracy.

[0079] S106, Increase the number of model iterations.

[0080] The initial number of iterations for the model is 0. After obtaining the updated intermediate model, the number of iterations is incremented by 1.

[0081] S107, determine whether the number of iterations of the model has reached the set number of sparse iterations n; if yes, proceed to step S108; if no, return to step S103.

[0082] If the number of iterations of the model has not reached the set number of sparse iterations n, it means that the set target sparsity has not been achieved, and the process returns to step S103 to continue the next round of iterations.

[0083] S108, use the currently obtained intermediate model as the target model.

[0084] If the model's iteration count has not reached the set sparse iteration count n, it indicates that the target sparsity has been achieved, and the current intermediate model can be used as the target model. The target model is a high-precision sparse large model, which includes fine-tuned hierarchical adaptive parameters and scaling bias parameters. Through the above process, while ensuring a high proportion of sparsity in the dense large model, the model accuracy can be efficiently improved.

[0085] The model sparsity method provided in this application can set different layer sparsity rates for different network layers based on their importance. By performing model sparsity processing on the neural network model multiple times, each time using a different model sparsity rate, a multi-round progressive iterative sparsity process is formed. After each model sparsity process, the auxiliary parameters in the neural network model are fine-tuned to finally obtain the target model. This reduces the resource consumption of large models, ensures the accuracy of the sparsified large model at low cost, alleviates the deployment cost of large models, and improves the inference competitiveness of large models. This application embodiment can achieve a compression of 3.3 to 5 times for large models containing tens of billions of parameters, while effectively ensuring the accuracy recovery of the large model after sparsity, improving the model accuracy by more than 15% in subsequent inference tasks.

[0086] Furthermore, this embodiment controls the model sparsity rate at each iteration step, progressively sparsifying the neural network model from large to small sparsity rates. This reduces the sudden drop in model accuracy caused by parameter mutations. Simultaneously, auxiliary parameters are used to dynamically adjust each network layer, restoring model accuracy at the layer level. During model sparsification and accuracy restoration, the importance of each layer in the neural network model is dynamically assessed. Highly important layers have less redundancy, so their sparsity is reduced. For less important layers, the sparsity is appropriately increased to remove redundant parameters and reduce the impact of important parameters on the model. Using different layer sparsity rates for different network layers effectively ensures the accuracy of the model after sparsification. Throughout the process, redundant parameters are sparsified as much as possible, with the sparsity increasing gradually, restoring model accuracy at the layer level. The aim is to minimize model accuracy loss while maintaining the target sparsity.

[0087] The target model obtained in this application embodiment can be used for practical reasoning, such as speech recognition and human-computer dialogue. For example, the target model can be applied to mobile devices and embedded systems. In resource-constrained environments such as smartphones and IoT devices, model sparsity can significantly reduce the size of neural network models, reduce memory usage and energy consumption, enabling complex neural network models to run in real time on these devices, such as speech recognition and image classification applications on mobile phones. The target model can also be applied to the field of autonomous driving. During the operation of an autonomous vehicle, sensors interact with the external environment and return relevant information such as roads, pedestrians, and obstacles to the relevant decision-making system. The decision-making system needs to make choices and decisions in a short time to determine the next trajectory of the car. At this time, model sparsity can reduce the size of the neural network model in the decision-making system, speed up the system decision-making process, and reduce hardware costs and power consumption, which is especially important for the limited computing resources in the vehicle. In addition to mobile devices and embedded systems, autonomous driving, the target model can also be applied to application scenarios that require compressed model memory and faster inference speed, such as intelligent robots and medical image analysis.

[0088] This application also provides an electronic device that can be used to implement the above. Figure 1 The method embodiments shown can achieve the beneficial effects of the above-described method embodiments by providing the functionality of the methods described above.

[0089] In some embodiments, the structure of the electronic device 500 can be as follows: Figure 5 As shown, the system includes a processor 501 and a memory 502 connected to the processor 501. The processor 501 and the off-chip memory 502 can be interconnected via a bus. The processor 501 can be a general-purpose processor, such as a microprocessor, or other conventional processor. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be a control bus, a system bus, or a data bus.

[0090] The memory 502 can be used to store the model parameters of the neural network model of the processor 501, as well as the data generated and used during the execution of the model sparsity method. The memory 502 can also be used to store software programs and modules. The processor 501 executes various functional applications and data processing of the electronic device 500, such as the model sparsity method provided in the embodiments of this application, by running the software programs and modules stored in the memory 502.

[0091] The memory 502 may include a program storage area and a data storage area. The program storage area may store the operating system, application programs of at least one application, etc.; the data storage area may be used to store user data, intermediate data generated during image rendering, etc. In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0092] The processor 501 in the electronic device 500 is used to run computer instructions or programs stored in the memory 502 to perform the functions described in any of the above method embodiments. In some embodiments, the processor 501 may include one or more processing units, such as a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), image signal processing unit (ISP), microprocessor (MP), controller, video codec, baseband processor, embedded neural network processing unit (NPU) in the field of AI, or digital signal processor (DSP), etc. Different processors may be independent devices, such as independent chips, or they may be integrated into the same chip. Different processing units may be independent devices or they may be integrated into one or more processors. The processor 501 may also include a controller, which can generate operation control signals according to the instruction opcode and timing signals to control the fetching and execution of instructions.

[0093] In some embodiments, the processor 501 can add auxiliary parameters to the original model to obtain an intermediate model. During the sparsification process of the intermediate model, the auxiliary parameters are adjusted to obtain the target model. The accuracy of the neural network model can be adjusted by adjusting the auxiliary parameters. It is not necessary to adjust all the parameters of the sparsified neural network model, which reduces the number of parameters that need to be adjusted. This reduces the computational resources consumed in the model sparsification process, speeds up the model sparsification process, and improves the efficiency of the model sparsification process.

[0094] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0095] This application also provides a model sparsity device, which can be applied to an electronic device, such as a computer, tablet computer, laptop computer, vehicle terminal, or server, such as a cloud server. In some embodiments, such as Figure 6 As shown, the model sparsity device 600 may include a parameter addition module 601 and a parameter adjustment module 602. The model sparsity device 600 can be used to implement... Figure 1 The method embodiments shown can achieve the beneficial effects of the above-described method embodiments by providing the functionality of the methods described above.

[0096] The parameter addition module 601 can be used to add auxiliary parameters to the original model to obtain an intermediate model; the auxiliary parameters are used to adjust the accuracy of the intermediate model. The parameter adjustment module 602 can be used to adjust the auxiliary parameters during the sparsification process of the intermediate model to obtain the target model.

[0097] In some embodiments, the parameter adjustment module 602 can be specifically used to determine the model sparsity rate for each sparsity processing based on a set target sparsity and a set number of sparsity iterations N; to perform N sparsity processing operations on the intermediate model according to the model sparsity rate for each sparsity processing operation; and to adjust auxiliary parameters for each sparsity processing operation. For example,

[0098] In some embodiments, the parameter adjustment module 602 can be specifically used to determine the layer sparsity of each network layer according to the model sparsity of this sparsity processing and the weight matrix of each network layer in the multiple network layers of the intermediate model during each sparsity processing; remove some parameters in each network layer according to the layer sparsity of each network layer to obtain the sparse intermediate model; and adjust the auxiliary parameters in the sparse intermediate model using a validation dataset.

[0099] In some embodiments, the parameter adjustment module 602 may include a progressive sparsity module, which can be used to determine the model sparsity rate for each sparsity processing based on a set target sparsity and a set number of sparsity iterations N; and perform N sparsity processing on the intermediate model according to the model sparsity rate of each sparsity processing. The progressive sparsity module may include a non-uniform sparsity module, which can be used to determine the layer sparsity rate of each network layer in each sparsity processing process based on the model sparsity rate of this sparsity processing and the weight matrix of each network layer in the multiple network layers of the intermediate model; remove some parameters in each network layer according to the layer sparsity rate of each network layer to obtain the sparsed intermediate model; and adjust the auxiliary parameters in the sparsed intermediate model using a validation dataset.

[0100] It should be noted that, in some embodiments, the parameter adding module 601 can be used to execute any step in the model sparsity method, and the parameter adjusting module 602 can also be used to execute any step in the model sparsity method. The steps implemented by the parameter adding module 601 and the parameter adjusting module 602 can be specified as needed. The parameter adding module 601 and the parameter adjusting module 602 respectively implement different steps in the model sparsity method to achieve all the functions of the model sparsity device. The model sparsity device 600 can also use more or fewer functional modules to implement the functions of the model sparsity device 600.

[0101] In the embodiments of this application, the functional modules can be integrated into a single processor, or each module can exist physically separately, or two or more modules can be integrated into a single module. The integrated modules can be implemented in hardware or as software functional modules.

[0102] This application also provides a chip that can be used to implement the functions of the above method embodiments, and thus can achieve the beneficial effects of the above method embodiments.

[0103] In some embodiments, the structure of the chip 700 can be as follows: Figure 7As shown, the device includes a processor 701 and a power supply circuit 702 connected to the processor 701. The processor 701 and the power supply circuit 702 can be interconnected via a bus. The processor 701 can be a digital signal processor (DSP), ASIC, field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or other specific integrated circuits. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be categorized as an address bus, data bus, control bus, etc. The power supply circuit 702 supplies power to the processor 701 via the bus.

[0104] The processor 701 can be connected to a memory located outside the chip or to a memory located inside the chip, and run software programs and modules stored in the memory to perform various functional applications and data processing of the chip 700, such as the model sparsity method provided in the embodiments of this application, so as to improve the accuracy and performance of the neural network model after sparsification and improve the efficiency of the model sparsification process.

[0105] In some embodiments, processor 701 may include one or more processing units, which may be independent devices or integrated into one or more processors. Processor 701 may also include a controller, which can generate operation control signals according to instruction opcodes and timing signals to control instruction fetching and execution.

[0106] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing a computer program or instructions. The computer program or instructions can constitute a computer program product.

[0107] This application also provides a computer program product comprising computer-executable instructions. In one embodiment, the computer-executable instructions are used to cause a computer to perform the functions described in the method embodiments above.

[0108] Computer-executable instructions can be stored in a computer-readable storage medium. This application also provides a computer-readable storage medium storing executable instructions. In one embodiment, the computer-executable instructions are used to cause a computer to perform the functions described in the method embodiments above.

[0109] The computer-readable storage medium provided in the embodiments of this application may be random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), register, hard disk, portable hard disk, CD-ROM, or any other form of computer-readable storage medium known in the art.

[0110] Computer-executable instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive.

[0111] One or more of the above modules or units can be implemented by software, hardware, or a combination of both. When any of the above modules or units is implemented by software, the software exists as computer program instructions and is stored in memory. The processor can be used to execute the program instructions and implement the above method flow. The processor can include, but is not limited to, at least one of the following: CPU, microprocessor, digital signal processor (DSP), microcontroller unit (MCU), or artificial intelligence processor, etc., various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor can be built into a SoC, DPU, or ASIC, or it can be a separate semiconductor chip. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as FPGAs, PLDs, or logic circuits that implement dedicated logic operations.

[0112] When the above modules or units are implemented in hardware, the hardware can be any one or any combination of CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, special purpose digital circuit, hardware accelerator or non-integrated discrete device, which can run the necessary software or perform the above method flow independently of software.

[0113] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application.

Claims

1. A model sparsity method, characterized in that, The method includes: Auxiliary parameters are added to the original model to obtain an intermediate model; the auxiliary parameters are used to adjust the accuracy of the intermediate model. During the sparsification process of the intermediate model, the auxiliary parameters are adjusted to obtain the target model.

2. The method according to claim 1, characterized in that, The process of adjusting the auxiliary parameters during the sparsification of the intermediate model includes: Based on the set target sparsity and the set number of sparsity iterations N, the model sparsity rate for each sparsity processing is determined. The intermediate model is subjected to N sparsification processes based on the model sparsity rate of each sparsification process; the auxiliary parameters are adjusted for each sparsification process.

3. The method according to claim 2, characterized in that, The intermediate model comprises multiple network layers; each of the N sparse processing operations includes the following steps: Based on the sparsity of the model in this sparsity processing and the weight matrix of each network layer in multiple network layers, the layer sparsity of each network layer is determined respectively. Based on the sparsity of each network layer, some parameters in each network layer are removed to obtain a sparse intermediate model. The auxiliary parameters in the sparse intermediate model are adjusted using a validation dataset.

4. The method according to claim 3, characterized in that, The step of adjusting the auxiliary parameters in the sparsed intermediate model using a validation dataset includes: Using the sparsed intermediate model as the model to be updated, repeat the following steps: The data in the validation dataset is input into the model to be updated to obtain the prediction result output by the model to be updated; The predicted result is compared with the reference result to obtain the loss value; Adjust the auxiliary parameters in the model to be updated based on the obtained loss value to obtain a new model to be updated, and return to the step of inputting the data in the validation dataset into the model to be updated until the obtained loss value converges.

5. The method according to any one of claims 2 to 4, characterized in that, The sparsity of the model gradually decreases from the first sparsification to the Nth sparsification.

6. The method according to any one of claims 1 to 5, characterized in that, The auxiliary parameters include hierarchical adaptation parameters added before each network layer of the original model and scaling bias parameters added to each network layer of the original model.

7. The method according to any one of claims 1 to 6, characterized in that, The original model is a large, pre-trained language model.

8. A model sparsity device, characterized in that, The device includes: The parameter addition module is used to add auxiliary parameters to the original model to obtain an intermediate model; the auxiliary parameters are used to adjust the accuracy of the intermediate model. The parameter adjustment module is used to adjust the auxiliary parameters during the sparsification process of the intermediate model to obtain the target model.

9. The apparatus according to claim 8, characterized in that, The parameter adjustment module is specifically used for: Based on the set target sparsity and the set number of sparsity iterations N, the model sparsity rate for each sparsity processing is determined. Based on the model sparsity rate of each sparsity processing step, the intermediate model is subjected to N sparsity processing steps. The auxiliary parameters are adjusted for each sparse processing step.

10. The apparatus according to claim 9, characterized in that, The intermediate model includes multiple network layers; the parameter adjustment module is specifically used for: In each sparsity processing process, the layer sparsity of each network layer is determined based on the model sparsity of this sparsity processing and the weight matrix of each network layer in multiple network layers. Based on the sparsity of each network layer, some parameters in each network layer are removed to obtain a sparse intermediate model. The auxiliary parameters in the sparse intermediate model are adjusted using a validation dataset.

11. An electronic device, characterized in that, The device includes at least one processor and at least one memory; wherein the one or more memories store one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 7.

12. A chip, characterized in that, It includes a processor and a power supply circuit; the power supply circuit is used to supply power to the processor, and the processor is used to execute a computer program to implement the method as described in any one of claims 1 to 7.

13. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 7.

14. A computer program product, characterized in that, It includes computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 7.

Images