Method and device for performing 2: 4 sparse pre-training by using continuous pruning function
By using continuous pruning function and weight scaling factor in deep neural networks, 2:4 sparse pre-training for deep neural networks is achieved, solving the optimization difficulties and accuracy reduction problems caused by non-continuous pruning functions in the existing technology. , improves the sparsity and efficiency of the model.
Patent Information
- Application Number
- CN202411114179.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-12-31
AI Technical Summary
In the prior art, the pruning function of the deep neural network model is discontinuous, resulting in difficulty in optimization and reduced accuracy, making it difficult to prune the model without reducing the accuracy of the model operation.
The continuous pruning function is used for 2:4 sparse pre-training. By chunking and pruning the original weights of the deep neural network, the sparse weights are obtained, and the model is optimized using the weight scaling factor to ensure that the model is propagated forward. Perform a fixed-scale pruning operation, complying with 2:4 sparse mode.
While maintaining high computing accuracy, the storage space and computing complexity of the model are reduced, and the sparsity and efficiency of the model are improved.
Smart Images
Figure CN119227766A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a method and device for performing 2:4 sparse pre-training using a continuous pruning function. Background Art
[0002] In the era of big data, massive amounts of data have accelerated the development of artificial intelligence, especially the application of deep neural networks. Using deep neural networks to mine potential features hidden in data through big data saves the early data preprocessing process. Although researchers have achieved good results by continuously increasing the number of deep neural network layers, it also makes the deep neural network model larger and larger, with more and more parameters, and the system resources used by the deep neural network model are greatly consumed.
[0003] Forward propagation is a key step in the neural network training process. It calculates and stores the intermediate variables and final output of the model in the order from the input layer to the output layer. In the prior art, many researchers have proposed methods for model pruning. The pruning function is a discontinuous function, and the commonly used optimization algorithms are based on first-order and second-order gradients. The premise for optimization is that the gradient exists and the function needs to satisfy the continuous characteristics. Therefore, the existing discontinuous functions will make optimization difficult and reduce the accuracy.
[0004] How to prune the model without reducing the model's calculation accuracy is a technical problem that needs to be solved at present. Summary of the invention
[0005] The present invention provides a method and device for performing 2:4 sparse pre-training using a continuous pruning function, so as to solve the defects existing in the prior art.
[0006] The present invention provides a method for performing 2:4 sparse pre-training using a continuous pruning function, comprising the following steps: Obtaining the original weights of the target model; wherein the target model is a deep neural network or a large model with a linear layer; Based on the original weights, a fixed-ratio pruning operation is performed before the target model is forward propagated to obtain sparse weights; The pre-acquired training data is input into the target model, and layer-by-layer calculation is performed according to the sparse weights and the hierarchical structure of the target model to obtain an output result; wherein the training data includes text data, image data or video data.
[0007] According to a method for performing 2:4 sparse pre-training using a continuous pruning function provided by the present invention, a fixed-ratio pruning operation is performed based on the original weights before the forward propagation of the target model to obtain sparse weights, which is implemented by the following formula:
[0008] in, represents sparse weights, represents the original weight, , is an n-dimensional vector, represents the weight scaling factor, represents the pruning function, express and The compound.
[0009] According to a method for performing 2:4 sparse pre-training using a continuous pruning function provided by the present invention, a fixed-ratio pruning operation is performed based on the original weights before the forward propagation of the target model to obtain sparse weights, including: The original weights are processed in blocks, and a fixed-ratio pruning operation is performed on the original weights after the block processing before each forward propagation of the target model to obtain initial sparse weights; The sparse weight is calculated based on the initial sparse weight and a pre-calculated weight scaling factor.
[0010] According to a method for performing 2:4 sparse pre-training using a continuous pruning function provided by the present invention, the The pruning function is expressed by the following formula:
[0011] in, It represents the small block formed after the original weight block processing, the size is 4, and t represents The absolute value of the third largest absolute value, express The i-th element in express The i-th element after pruning.
[0012] According to a method for performing 2:4 sparse pre-training using a continuous pruning function provided by the present invention, a fixed-ratio pruning operation is performed on the original weights after block processing before forward propagation of the target model, comprising: for The first and second largest ones in the pruning operation, the absolute value of the first and second largest ones after the pruning operation are both the difference between the original absolute value and t, and the sign does not change; for The third and fourth largest ones are mapped to 0.
[0013] According to a method for performing 2:4 sparse pre-training using a continuous pruning function provided by the present invention, the weight scaling factor is implemented by the following formula:
[0014] in, express The transpose of .
[0015] The present invention also provides a device for performing 2:4 sparse pre-training using a continuous pruning function, comprising the following modules: An acquisition module, used to acquire the original weights of a target model; wherein the target model is a deep neural network or a large model with a linear layer; A pruning module, used for performing a fixed-ratio pruning operation based on the original weights before forward propagation of the target model to obtain sparse weights; A training module is used to input pre-acquired training data into the target model, perform layer-by-layer calculations according to the sparse weights and the hierarchical structure of the target model, and obtain output results; wherein the training data includes text data, image data or video data.
[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for performing 2:4 sparse pre-training using a continuous pruning function as described in any one of the above is implemented.
[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for performing 2:4 sparse pre-training using a continuous pruning function as described in any of the above.
[0018] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for performing 2:4 sparse pre-training using a continuous pruning function as described in any one of the above methods.
[0019] The present invention provides a method and device for 2:4 sparse pre-training using a continuous pruning function, by obtaining the original weights of a target model; wherein the target model is a deep neural network or a large model with a linear layer; based on the original weights, a fixed-ratio pruning operation is performed before the forward propagation of the target model to obtain sparse weights; the pre-acquired training data is input into the target model, and layer-by-layer calculations are performed according to the sparse weights and the hierarchical structure of the target model to obtain output results; wherein the training data includes text data, image data or video data. It can be seen from this that the present invention obtains sparse weights by performing pruning operations before the model is forward propagated, and uses the sparse weights to complete forward propagation, thereby maintaining a high degree of accuracy while optimizing the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0021] Figure 1 It is a flow chart of the method for performing 2:4 sparse pre-training using a continuous pruning function provided by the present invention.
[0022] Figure 2 It is a structural schematic diagram of a device for performing 2:4 sparse pre-training using a continuous pruning function provided by the present invention.
[0023] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0025] Before describing the present invention, the relevant terms involved in the present invention are first explained.
[0026] 2:4 semistructured sparsity. A new sparsity pattern. A matrix that conforms to 2:4 semistructured sparsity must satisfy the following property: every group of 4 consecutive elements in the matrix must have at least two zeros in each group. On newer Nvidia GPU hardware, matrix multiplication can theoretically be twice as fast if one of the operands in the matrix multiplication conforms to the 2:4 semistructured sparsity pattern.
[0027] 2:4 sparse training: 2:4 sparse training. In order to obtain a sparse neural network with weights sparse according to the 2:4 semi-structured sparse pattern, the network is pruned when optimizing the weights in the pre-training stage.
[0028] Soft-thresholding: Soft thresholding. Contrary to hard-thresholding. During the model pruning process, hard thresholding resets the weights below the threshold to zero, and the weights above the threshold remain unchanged. Soft thresholding adjusts this process to make it smoother and have better continuity.
[0029] Weight-rescaling: Weight rescaling is a step in the deep learning optimization process, which refers to scaling the weights according to a certain ratio during the training process before performing forward and backward propagation. This is usually to improve the convergence speed, stability or performance of the model.
[0030] MSE (Mean Squared Error): Mean square error is a common method to measure the error between the weights after pruning and the weights before pruning. It is obtained by calculating the average of the squares of the difference between the predicted value and the true value. The smaller the MSE value, the smaller the error.
[0031] SR-STE (Sparse Refined Straight-Through Estimator) is an improvement to the optimization algorithm in sparse neural network training. It aims to reduce the error of gradient approximation and improve training efficiency and model performance by adding a decay term (although the weights have been set to zero, a decay term is used to reduce their impact in back propagation).
[0032] Forward propagation: It is a key step in the neural network training process. It calculates and stores the activation values and final outputs of the model's intermediate layers in the order from the input layer to the output layer. The results of forward propagation are used to calculate the loss function and optimize the model's parameters through back propagation.
[0033] Back propagation: It is an important algorithm used in neural networks to train and optimize model parameters. It mainly calculates the gradient of the loss function to the model parameters and backpropagates these gradients from the output layer to the input layer to update the model weights, thereby minimizing the value of the loss function and improving the prediction performance of the model.
[0034] Model Pruning: Also known as model sparsification, it is a model compression technique that removes "unnecessary" weights or biases from a neural network. The core idea of pruning technology is to reduce the storage space and computational complexity of the model as much as possible while maintaining the accuracy of the model.
[0035] Combine the following Figure 1-Figure 3 The present invention describes a method and device for 2:4 sparse pre-training using a continuous pruning function.
[0036] Figure 1 is a flow chart of a method for performing 2:4 sparse pre-training using a continuous pruning function provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 100: Obtain the original weights of the target model; wherein the target model is a deep neural network or a large model with a linear layer.
[0037] It should be noted that the present invention is aimed at the linear layer in the model, so any model with a linear layer can be applied. For example, large language models such as GPT (Generative Pre-trained Transformer), OPT (Outer Product of Tensors), Megatron, LLaMA, Phi, Qwen, etc. can be applied. In addition, it can also be applied to most machine vision models, such as DeiT (Data-efficient image Transformers), ViT (Vision Transformer), Diffusion, etc.
[0038] Step 200: Based on the original weights, a fixed-ratio pruning operation is performed before the target model is forward propagated to obtain sparse weights.
[0039] Specifically, based on the original weights, a fixed-ratio pruning operation is performed before the target model is forward propagated to obtain sparse weights, which is achieved by the following formula.
[0040]
[0041] in, represents sparse weights, represents the original weight, , is an n-dimensional vector, represents the weight scaling factor, represents the pruning function, express and The compound.
[0042] The embodiment of the present invention proposes the above-mentioned continuous and sparse mapping function to replace the pruning process of SR-STE, so that the model maintains higher calculation accuracy.
[0043] It should be noted that the 2:4 sparse mode is a sparsity technology used to compress and accelerate models in deep neural networks (DNNs). Specifically, it means that in a matrix or weight tensor, 2 out of every 4 elements are non-zero, that is, the sparsity ratio is 2:4 or 50%, that is, the fixed ratio is 2:4 or 50%.
[0044] Through the above formula, pruning is performed before forward propagation so that the sparse weights conform to the 2:4 sparse pattern.
[0045] Specifically, the present invention mainly involves two optimization parts before the model forward propagation: soft-thresholding and weight-rescaling, which are described as follows.
[0046] Soft-thresholding part: The purpose of this part is to make the optimized function space more continuous. Specifically: The original weights are processed in blocks, and a fixed-ratio pruning operation is performed on the original weights after the block processing before each forward propagation of the target model to obtain initial sparse weights; The sparse weight is calculated based on the initial sparse weight and a pre-calculated weight scaling factor.
[0047] In one embodiment, it is achieved by the following formula.
[0048]
[0049] in, It represents the small block formed after the original weight block processing, the size is 4, and t represents The absolute value of the third largest absolute value, express The i-th element in express The i-th element after pruning.
[0050] It should be noted that w is the original weight, that is, an m×n matrix, where n is usually a multiple of 4. Represents a small block of 4 adjacent elements in the original weight.
[0051] Specifically, for The first and second largest of the two, after the pruning operation, the absolute value of the first and second largest are the difference between the original absolute value and t, and the sign does not change; The third and fourth largest ones are mapped to 0.
[0052] Weight-rescaling part: The purpose of this part is to make the output activation value obtained by the sparse weight as close as possible to the output activation value obtained by the original weight. It contains two key points: fixing the weight scaling factor and minimizing the mean square error.
[0053] Fixed weight scaling factor: calculated once when the model is initialized , and remains unchanged thereafter.
[0054] Minimize mean square error: Calculate The purpose of is to minimize the Euclidean distance between the sparse weights and the original weights, which is achieved by the following formula.
[0055]
[0056] Among them, MSE represents mean square error, express The transpose of .
[0057] Step 300: input the pre-acquired training data into the target model, perform layer-by-layer calculation according to the sparse weights and the hierarchical structure of the target model, and obtain an output result; wherein the training data includes text data, image data or video data.
[0058] It should be noted that the training data for large language models are usually large-scale text corpora, which contain a large amount of text data and are used to train large language models, such as OpenAI's GPT series and Google's PaLM. The training data for machine vision models are mainly image and video datasets, which are used to train models to recognize and process objects and scenes in images.
[0059] Specifically, first, the training data (such as images, text, etc.) is converted into a format that the target model can process, such as a vector or matrix; the training data is input into the target model, and the input data is calculated layer by layer through weights, biases, and activation functions to finally obtain the output result; under sparse weight control, since most of the weights are zero, the calculation process will become more efficient.
[0060] Furthermore, STE is used in the back propagation of the model, that is, the gradient of the sparse weights is regarded as the gradient of the original weights, and the gradient of the sparse weights is used to update the original weights.
[0061] The above is a description of the steps of the method for 2:4 sparse pre-training using a continuous pruning function provided by the present invention. From the description of the above steps, it can be seen that according to the method for 2:4 sparse pre-training using a continuous pruning function provided by the present invention, the original weights of the target model are obtained; wherein the target model is a deep neural network or a large model with a linear layer; based on the original weights, a fixed ratio of pruning operations are performed before the forward propagation of the target model to obtain sparse weights; the pre-acquired training data is input into the target model, and the sparse weights and the hierarchical structure of the target model are calculated layer by layer to obtain the output result; wherein the training data includes text data, image data or video data. It can be seen from this that the present invention obtains sparse weights by performing a pruning operation before the model is forward propagated, and uses the sparse weights to complete the forward propagation, thereby maintaining a high accuracy while optimizing the model.
[0062] The following is a description of the device for performing 2:4 sparse pre-training using a continuous pruning function provided by the present invention. The device for performing 2:4 sparse pre-training using a continuous pruning function described below and the method for performing 2:4 sparse pre-training using a continuous pruning function described above can be referenced to each other.
[0063] Figure 2 is a schematic diagram of the structure of the device for performing 2:4 sparse pre-training using a continuous pruning function provided by the present invention, such as Figure 2 As shown, the device for performing 2:4 sparse pre-training using a continuous pruning function provided by the present invention includes: An acquisition module 201 is used to acquire the original weights of a target model; wherein the target model is a deep neural network or a large model with a linear layer; A pruning module 202, configured to perform a fixed-ratio pruning operation based on the original weights before forward propagation of the target model to obtain sparse weights; The training module 203 is used to input the pre-acquired training data into the target model, perform layer-by-layer calculation according to the sparse weights and the hierarchical structure of the target model, and obtain an output result; wherein the training data includes text data, image data or video data.
[0064] The device for 2:4 sparse pre-training using a continuous pruning function provided by the present invention obtains the original weights of a target model; wherein the target model is a deep neural network or a large model with a linear layer; based on the original weights, a fixed-ratio pruning operation is performed before the forward propagation of the target model to obtain sparse weights; the pre-acquired training data is input into the target model, and layer-by-layer calculations are performed according to the sparse weights and the hierarchical structure of the target model to obtain output results; wherein the training data includes text data, image data or video data. It can be seen from this that the present invention obtains sparse weights by performing a pruning operation before the model is forward propagated, and uses the sparse weights to complete forward propagation, thereby maintaining a high degree of accuracy while optimizing the model.
[0065] Based on the above embodiment, in this embodiment, based on the original weights, a fixed-ratio pruning operation is performed before the forward propagation of the target model to obtain sparse weights, which is implemented by the following formula:
[0066] in, represents sparse weights, represents the original weight, , is an n-dimensional vector, represents the weight scaling factor, represents the pruning function, express and The compound.
[0067] Based on the above embodiment, in this embodiment, the pruning module 202 is specifically used for: The original weights are processed in blocks, and a fixed-ratio pruning operation is performed on the original weights after the block processing before each forward propagation of the target model to obtain initial sparse weights; The sparse weight is calculated based on the initial sparse weight and a pre-calculated weight scaling factor.
[0068] Based on the above embodiments, in this embodiment, the The pruning function is expressed by the following formula:
[0069] in, It represents the small block formed after the original weight block processing, the size is 4, and t represents The absolute value of the third largest absolute value, express The i-th element in express The i-th element after pruning.
[0070] Based on the above embodiment, in this embodiment, the pruning module 202 is specifically used for: for The first and second largest ones in the pruning operation, the absolute value of the first and second largest ones after the pruning operation are both the difference between the original absolute value and t, and the sign does not change; for The third and fourth largest ones are mapped to 0.
[0071] Based on the above embodiment, in this embodiment, the weight scaling factor is implemented by the following formula:
[0072] in, express The transpose of .
[0073] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the method for performing 2:4 sparse pre-training using a continuous pruning function, the method comprising: Obtaining the original weights of the target model; wherein the target model is a deep neural network or a large model with a linear layer; Based on the original weights, a fixed-ratio pruning operation is performed before the target model is forward propagated to obtain sparse weights; The pre-acquired training data is input into the target model, and layer-by-layer calculation is performed according to the sparse weights and the hierarchical structure of the target model to obtain an output result; wherein the training data includes text data, image data or video data.
[0074] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0075] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the method for performing 2:4 sparse pre-training using a continuous pruning function provided by the above methods, the method comprising: Obtaining the original weights of the target model; wherein the target model is a deep neural network or a large model with a linear layer; Based on the original weights, a fixed-ratio pruning operation is performed before the target model is forward propagated to obtain sparse weights; The pre-acquired training data is input into the target model, and layer-by-layer calculation is performed according to the sparse weights and the hierarchical structure of the target model to obtain an output result; wherein the training data includes text data, image data or video data.
[0076] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the method for performing 2:4 sparse pre-training using a continuous pruning function provided by the above methods, the method comprising: Obtaining the original weights of the target model; wherein the target model is a deep neural network or a large model with a linear layer; Based on the original weights, a fixed-ratio pruning operation is performed before the target model is forward propagated to obtain sparse weights; The pre-acquired training data is input into the target model, and layer-by-layer calculation is performed according to the sparse weights and the hierarchical structure of the target model to obtain an output result; wherein the training data includes text data, image data or video data.
[0077] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.
[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for 2:4 sparse pre-training using a continuous pruning function, characterized in that: include: Obtaining the original weights of the target model; wherein the target model is a deep neural network or a large model with a linear layer; Based on the original weights, a fixed-ratio pruning operation is performed before the target model is forward propagated to obtain sparse weights; The pre-acquired training data is input into the target model, and layer-by-layer calculation is performed according to the sparse weights and the hierarchical structure of the target model to obtain an output result; wherein the training data includes text data, image data or video data.
2. The method for 2:4 sparse pre-training using a continuous pruning function according to claim 1, characterized in that: Based on the original weights, a fixed-ratio pruning operation is performed before the target model is forward propagated to obtain sparse weights, which is implemented by the following formula: in, represents sparse weights, represents the original weight, , is an n-dimensional vector, represents the weight scaling factor, represents the pruning function, express and The compound.
3. The method for 2:4 sparse pre-training using a continuous pruning function according to claim 1, characterized in that: The method of performing a fixed-ratio pruning operation based on the original weights before forward propagation of the target model to obtain sparse weights includes: The original weights are processed in blocks, and a fixed-ratio pruning operation is performed on the original weights after the block processing before each forward propagation of the target model to obtain initial sparse weights; The sparse weight is calculated based on the initial sparse weight and a pre-calculated weight scaling factor.
4. The method for 2:4 sparse pre-training using a continuous pruning function according to claim 3, characterized in that: Said The pruning function is expressed by the following formula: in, It represents the small block formed after the original weight block processing, the size is 4, and t represents The absolute value of the third largest absolute value, express The i-th element in express The i-th element after pruning.
5. The method for 2:4 sparse pre-training using a continuous pruning function according to claim 4, characterized in that: The method of performing a fixed-ratio pruning operation on the original weights after block processing before forward propagation of the target model includes: for The first and second largest ones in the pruning operation, the absolute value of the first and second largest ones after the pruning operation are both the difference between the original absolute value and t, and the sign does not change; for The third and fourth largest ones are mapped to 0.
6. The method for 2:4 sparse pre-training using a continuous pruning function according to claim 3, characterized in that: The weight scaling factor is implemented by the following formula: in, express The transpose of .
7. A device for 2:4 sparse pre-training using a continuous pruning function, characterized in that: include: An acquisition module, used to acquire the original weights of a target model; wherein the target model is a deep neural network or a large model with a linear layer; A pruning module, used for performing a fixed-ratio pruning operation based on the original weights before forward propagation of the target model to obtain sparse weights; A training module is used to input pre-acquired training data into the target model, perform layer-by-layer calculations according to the sparse weights and the hierarchical structure of the target model, and obtain output results; wherein the training data includes text data, image data or video data.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for performing 2:4 sparse pre-training using a continuous pruning function as described in any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for performing 2:4 sparse pre-training using a continuous pruning function as described in any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for performing 2:4 sparse pre-training using a continuous pruning function as described in any one of claims 1 to 6 is implemented.