Data processing method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The application provides a data processing method and device, electronic equipment, computer readable storage medium and computer program product. The method comprises: obtaining to-be-processed data and a sparsity threshold set of a data processing model; inputting the to-be-processed data into the data processing model, performing multiple rounds of iterative processing on the to-be-processed data by the data processing model, and obtaining a first model output matrix; wherein, in at least one round of iterative processing, a mask matrix is determined based on a first activation feature vector output by an activation layer and a sparsity threshold, the mask matrix being used for sparse calculation in the current step of iterative processing and multiple steps of iterative processing after the current step; and a data processing result of the to-be-processed data is determined based on the first model output matrix. Through the application, the calculation efficiency of the data processing model can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and particularly relates to a data processing method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] In deep learning models, as the amount of data and model complexity continue to increase, the demand for computing resources also grows exponentially. This not only leads to an increase in the time-consuming of training and inference processes, but also puts higher requirements on hardware resources. Although sparsification can reduce unnecessary calculations and parameters in the model, effectively reducing the computational complexity and storage requirements of the model. However, sparsification technology may lead to a decrease in model accuracy in practical applications, and in the case of maintaining model accuracy, the model can only be sparsified to a limited extent, and the computational efficiency of the model is still low. SUMMARY

[0003] The embodiments of the present application provide a data processing method, device, electronic equipment, computer readable storage medium and computer program product, which can improve the computational efficiency of the data processing model.

[0004] The technical scheme of the embodiments of the present application is as follows:

[0005] The embodiments of the present application provide a data processing method, which comprises: obtaining to-be-processed data and a sparsity threshold set of a data processing model; wherein the data processing model comprises a plurality of data processing modules, each data processing module comprising a first linear layer, an activation layer and a second linear layer; the sparsity threshold set comprises: the sparsity threshold of each data processing module in the data processing model at each time step of each round of iterative processing; inputting the to-be-processed data into the data processing model, and performing multiple rounds of iterative processing on the to-be-processed data through the data processing model to obtain a first model output matrix; wherein, in at least one round of iterative processing, a mask matrix is determined based on a first activation feature vector output by the activation layer and the sparsity threshold, the mask matrix being used for sparse calculation in the current step of iterative processing and multiple steps of iterative processing after the current step; determining a data processing result of the to-be-processed data based on the first model output matrix.

[0006] The embodiment of the application provides a data processing device, comprising: a data acquisition module, configured to acquire to-be-processed data and a sparsity threshold set of a data processing model; wherein the data processing model comprises a plurality of data processing modules, each data processing module comprises a first linear layer, an activation layer and a second linear layer; the sparsity threshold set comprises a sparsity threshold of each data processing module in the data processing model at a corresponding time step in each round of iterative processing; a data processing module, configured to input the to-be-processed data into the data processing model, perform multiple rounds of iterative processing on the to-be-processed data through the data processing model, and obtain a first model output matrix; wherein in at least one round of iterative processing, a mask matrix is determined based on a first activation feature vector output by the activation layer and the sparsity threshold, and the mask matrix is used for sparse calculation in current step iterative processing and multiple steps of iterative processing after the current step; and a data output module, configured to determine a data processing result of the to-be-processed data based on the first model output matrix.

[0007] In the above scheme, the data acquisition module is further configured to acquire a channel importance score set of the data processing model; the channel importance score set comprises a channel importance score of each channel in the activation layer; the activation layer in the data processing model is rearranged based on the channel importance score of each channel, and a model parameter after channel rearrangement is obtained; and the inputting of the to-be-processed data into the data processing model, the multiple rounds of iterative processing of the to-be-processed data through the data processing model, and the obtaining of the first model output matrix comprise: inputting the to-be-processed data into a channel rearranged data processing model corresponding to the model parameter after channel rearrangement, performing multiple rounds of iterative processing on the to-be-processed data through the channel rearranged data processing model, and obtaining the first model output matrix.

[0008] In the above scheme, the data acquisition module is further configured to acquire first calibration data; the first calibration data is input into the data processing model, linear processing and activation processing of the first calibration data are sequentially performed through the first linear layer and the activation layer of the data processing model, and a second activation feature vector is obtained; an absolute average value of an element corresponding to each channel of the activation layer in the second activation feature vector is determined; the absolute average value of each channel is determined as a channel importance score of the channel, and the channel importance score set is obtained.

[0009] In the above scheme, the data acquisition module is further configured to acquire, for each activation layer in the data processing model, a weight parameter of each channel in the activation layer; sort channel feature vectors of the second activation feature vector in an order from small to large according to the channel importance scores to obtain an activation value channel rearrangement vector; and perform weight transformation on the weight parameter of each channel in the activation layer using the activation value channel rearrangement vector to obtain a model parameter after channel rearrangement.

[0010] In the above scheme, the data acquisition module is further configured to input second calibration data into the data processing model, perform N rounds of iterative processing on the second calibration data by using the data processing model to obtain a second model output matrix; acquire a candidate sparsity set; for the Kth round of iterative processing, perform sparsity processing on each data processing module in the data processing model according to each candidate sparsity in the candidate sparsity set to obtain a plurality of sparse models corresponding to each candidate sparsity; K is an integer greater than 0 and less than or equal to N; input the second calibration data into each sparse model to obtain a plurality of third model output matrices; and determine a sparsity threshold set of the data processing model from the candidate sparsity set based on the plurality of third model output matrices and the second model output matrix.

[0011] In the above scheme, the data acquisition module is further configured to determine a cosine similarity between each third model output matrix and the second model output matrix; determine a target sparsity of each data processing module in the Kth round of iterative processing from the candidate sparsity set based on the cosine similarity; and store the target sparsity as the sparsity threshold of the data processing module in the Kth round of iterative processing in the sparsity threshold set.

[0012] In the above scheme, the data acquisition module is further configured to, for each data processing module in the Kth round of iterative processing, in response to determining at least one target cosine similarity greater than a preset similarity threshold from all cosine similarities corresponding to candidate sparsities in the candidate sparsity set, determine a maximum candidate sparsity from at least one candidate sparsity corresponding to the at least one target cosine similarity as the target sparsity of the data processing module in the Kth round of iterative processing; and in response to not determining a target cosine similarity greater than a preset similarity threshold from all cosine similarities corresponding to candidate sparsities in the candidate sparsity set, set the target sparsity of the data processing module in the Kth round of iterative processing as a preset value.

[0013] In the above scheme, the data processing module is further configured to: respond to determining that the current step is an iterative update step based on the number of iteration steps of the current step; obtain the first activation feature vector output by the activation layer in the current step and the sparsity threshold of the data processing module where the activation layer is located in the current step; divide the first activation feature vector into multiple sub-blocks according to a preset block size; determine the importance score of each sub-block based on the elements in each sub-block; respond to any sub-block having an importance score greater than or equal to the sparsity threshold, determine the mask value corresponding to the sub-block in the mask matrix as the first mask value; respond to any sub-block having an importance score less than the sparsity threshold, determine the mask value corresponding to the sub-block in the mask matrix as the second mask value.

[0014] In the above scheme, the data processing module is further configured to, in response to determining that the current step is a non-iterative update step based on the number of iteration steps of the current step, obtain the mask matrix determined by the most recent iteration update step before the current step; and determine the mask matrix determined by the most recent iteration update step before the current step as the mask matrix of the current step.

[0015] In the above scheme, the data processing module is further configured to, in each iteration, perform a first linear processing on the input features based on the mask matrix of the data processing module, through the first linear layer of the data processing module, to obtain a first linear feature vector for each data processing module in the data processing model; wherein, for the first data processing module in the first iteration, the input feature is the data to be processed; for the first data processing module in the i-th iteration, the input feature is the model output matrix in the (i-1)-th iteration; for the j-th data processing module in the i-th iteration, the input feature is the model output matrix in the i-th iteration. The output feature vectors of j-1 data processing modules; where i is an integer greater than 1 and less than or equal to N; j is an integer greater than 1 and less than or equal to L, where L is the total number of data processing modules in the data processing model; the first linear feature vector is activated by the activation layer of the data processing module to obtain a third activated feature vector; based on the mask matrix of the data processing module, the third activated feature vector is processed by the second linear layer of the data processing module to obtain the output feature vector of the data processing module; the model output matrix of the last round of iteration is determined as the first model output matrix obtained after the multi-round iteration.

[0016] In the above scheme, the data processing module is further configured to: divide the input features into blocks according to a preset block size to obtain multiple input blocks; determine the mask value of each input block from the mask matrix of the data processing module; skip the weight calculation of the input block in response to the mask value of any input block being the second mask value; and perform a weighted summation of the input blocks based on the weight parameters of the input blocks in response to the mask value of any input block being the first mask value to obtain the first linear feature vector.

[0017] This application provides an electronic device, which includes: a memory for storing computer-executable instructions or computer programs; and a processor for executing the computer-executable instructions or computer programs stored in the memory to implement the data processing method provided in this application.

[0018] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data processing method provided in this application when executed by a processor.

[0019] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the data processing method provided in this application.

[0020] The embodiments of this application have the following beneficial effects:

[0021] During data processing, the sparsity threshold set of the data to be processed and the data processing model can be obtained first. This sparsity threshold set includes the sparsity threshold of each data processing module in the data processing model at each time step of each iteration. By pre-setting different sparsity thresholds for each data processing module at each time step, the sparsity of the data processing model can be improved. Then, the data to be processed is input into the data processing model, which performs multiple iterations to obtain the first model output matrix. In at least one iteration, a mask matrix is ​​determined based on the first activation feature vector output by the activation layer and the sparsity threshold. This mask matrix is ​​used for sparse computation in the current iteration and subsequent iterations. By determining the mask matrix, computation of the sparse parts can be skipped, reducing unnecessary calculations and improving model efficiency. Furthermore, the same mask matrix can be shared across multiple iterations, significantly reducing computational resource consumption in the mask matrix determination process. This ensures data processing accuracy while improving computational resource utilization. Finally, the data processing result is determined based on the output matrix of the first model. This improves both the sparsity of the data processing model and the computational efficiency of the model by using a mask matrix to achieve sparse computation. Attached Figure Description

[0022] Figure 1 This is an optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0023] Figure 2 This is another optional flowchart illustrating the data processing method provided in the embodiments of this application;

[0024] Figure 3 This is a schematic diagram of the implementation process for obtaining the sparsity threshold provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram illustrating the implementation process of obtaining channel importance scores provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram illustrating the implementation process of obtaining the model parameters after channel rearrangement provided in an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of the sparsity distribution of different time steps and blocks provided in the embodiments of this application;

[0028] Figure 7 This is a schematic diagram of the activation value mask distribution under different sparse methods provided in the embodiments of this application;

[0029] Figure 8This is a schematic diagram of the channel sorting provided in the embodiments of this application;

[0030] Figure 9 This is a schematic diagram illustrating the principle of straddle mask multiplexing provided in the embodiments of this application;

[0031] Figure 10 This is a structural block diagram of a data processing apparatus provided in an embodiment of this application;

[0032] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0036] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0037] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0038] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0039] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0040] 1) Responding to: used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0041] 2) Sparsity: Sparsity is a metric that measures the proportion of non-zero elements in data. Sparsity can be used to describe the proportion of zero elements in the weight matrix of a data processing module. The higher the sparsity, the higher the proportion of zero elements and the lower the proportion of non-zero elements in the data processing module.

[0042] 3) Importance score: A value used to measure the importance of each sub-block. The higher the importance score, the more important the sub-block is.

[0043] 4) Channel Importance Score: A value used to measure the importance of each channel. The higher the channel importance score, the more important the channel is.

[0044] To better understand the data processing method provided in the embodiments of this application, the data processing methods in related technologies will be described below.

[0045] In related technologies, to achieve contextual sparsity prediction, a sparsity predictor composed of multilayer perceptrons is introduced. This sparsity predictor itself uses additional weight parameters and requires retraining for each target model, resulting in significant training overhead and reducing the applicability and reusability of the method. Furthermore, during model inference, the sparsity predictor needs to dynamically infer the input features in real time, leading to additional computational burden and inference latency. This limits the speedup effect of sparse inference on actual end-to-end inference time and fails to fully unleash the performance potential of sparsity. Although structured pruning can be used to achieve low-cost sparsity without retraining, the pruning effect still has certain limitations: First, the pruning ratio is limited, typically only achieving a sparsity rate of about 10%, which is low and cannot achieve significant inference acceleration. Second, due to insufficient sparsity, the number of parameters and computational cost of the pruned model are still large, failing to fully unleash the performance potential brought by model compression and limiting the model's application in resource-constrained environments.

[0046] Based on the problems existing in related technologies, this application provides a data processing method. When processing data, the data to be processed and the sparsity threshold set of the data processing model can be obtained first. The sparsity threshold set includes the sparsity threshold of each data processing module in the data processing model at the time step corresponding to each round of iterative processing. By presetting different sparsity thresholds for each data processing module at each time step, the sparsity of the data processing model can be improved. Then, the data to be processed is input into the data processing model, which performs multiple rounds of iterative processing to obtain the first model output matrix. In at least one iteration, a mask matrix is ​​determined based on the first activation feature vector output by the activation layer and a sparsity threshold. This mask matrix is ​​used for sparse computation in the current iteration and subsequent iterations. By determining the mask matrix, computation of the sparse portion can be skipped, reducing unnecessary calculations and improving model efficiency. Furthermore, the same mask matrix can be shared across multiple iterations, significantly reducing computational resource consumption in the mask matrix determination process. This ensures data processing accuracy while improving computational resource utilization. Finally, the data processing result is determined based on the first model output matrix. Thus, the sparsity of the data processing model is improved, and sparse computation is achieved through the mask matrix, thereby enhancing the computational efficiency of the data processing model.

[0047] The data processing method provided in this application can be applied to electronic devices such as laptops, tablets, and desktop computers. This application does not impose any restrictions on the specific type of electronic device.

[0048] The data processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0049] Figure 1 This is an optional flowchart illustrating a data processing method provided in an embodiment of this application, which can be applied to electronic devices. The following description uses an electronic device as a server as an example. It should be noted that the data processing method provided in this embodiment can be applied at least to large generative models based on diffusion, with the aim of accelerating the inference efficiency of such models. Figure 1 As shown, the method includes the following steps S101 to S103:

[0050] Step S101: Obtain the data to be processed and the sparsity threshold set of the data processing model.

[0051] Here, the data processing model includes multiple data processing modules, each of which includes a first linear layer, an activation layer, and a second linear layer. The sparsity threshold set includes the sparsity threshold of each data processing module in the data processing model at each time step corresponding to each round of iteration.

[0052] In this embodiment, the data to be processed can be data of any form: images, text, videos, and audio, etc. It can be determined according to the specific task implemented by the data processing model. The specific task implemented by the data processing model can be a video generation task or an image generation task. For example, the task can be a text-to-video task or an image-to-video task within a video generation task, or it can be a text-to-image task or an image-to-image task within an image generation task. When the task is a text-to-video task, the data to be processed can be at least text data, and videos of different styles and types can be generated based on the input text data using the data processing method provided in this embodiment. When the task is an image-to-video task, the data to be processed can be at least image data, and videos of different styles and types can be generated based on the input image data using the data processing method provided in this embodiment. When the task is a text-to-image task, the data to be processed can be at least text data, and images of different styles and types can be generated based on the input text data using the data processing method provided in this embodiment. When the task is an image-to-image task, the data to be processed can be at least image data, and new images can be generated based on the input image data using the data processing method provided in this embodiment.

[0053] The data processing model can be a Transformer model, a convolutional neural network, a multimodal fusion model, etc., and this application embodiment is not limited to this; it can be selected according to the actual situation. The data processing model can include multiple data processing modules. Each data processing module can include a first linear layer, an activation layer, and a second linear layer. For example, the data processing model is a Transformer model, and the Transformer model can include multiple Transformer modules. Each Transformer module can include a first linear layer, an activation layer, and a second linear layer.

[0054] Sparsity is a metric that measures the proportion of non-zero elements in data. It can be used to describe the proportion of zero elements in the weight matrix of a data processing module. Higher sparsity indicates a higher proportion of zero elements and a lower proportion of non-zero elements in the data processing module. The sparsity threshold set includes the sparsity threshold for each data processing module in the data processing model at each time step of each iteration. In other words, each data processing module has a sparsity threshold at different time steps. At a given time step, the sparsity threshold for each data processing module can be the same or different. For the same data processing module, the sparsity threshold can be the same or different at different time steps.

[0055] Step S102: Input the data to be processed into the data processing model, and perform multiple rounds of iterative processing on the data to be processed through the data processing model to obtain the first model output matrix.

[0056] Here, during at least one round of iteration, a mask matrix is ​​determined based on the first activation feature vector output by the activation layer and the sparsity threshold. The mask matrix is ​​used for sparse computation during the current iteration and subsequent iterations.

[0057] In this embodiment, multi-round iterative processing refers to the process by which the data processing model processes the data to be processed multiple times. Each processing round is called an iteration. The first model output matrix can be the final output matrix generated by the data processing model after multiple rounds of iterative processing. This matrix can contain the features or results processed by the data processing model. For example, if the data to be processed is image data, the first model output matrix may be a probability distribution, which represents the probability that the input image belongs to different categories. An activation layer refers to a non-linear layer introduced into the data processing model. Common activation functions include ReLU, Sigmoid, Tanh, GELU, etc. The first activation feature vector refers to the feature vector output by the activation layer. For example, the feature vector after processing by the ReLU activation function in the feedforward network layer of the Transformer model. The mask matrix can be a matrix with the same shape as the first activation feature vector, used to mark whether the vector at each position in the first activation feature vector can skip the calculation (i.e., sparse computation). The values ​​in the mask matrix are usually 0 or 1, where 0 indicates that the vector at that position can skip the calculation, and 1 indicates that the vector at that position needs to be calculated.

[0058] In the process of iteratively processing the data to be processed by the data processing model, at least in one iteration, a mask matrix can be determined based on the first activation feature vector output by the activation layer and the sparsity threshold. The mask matrix is ​​used for sparse computation in the current iteration and in subsequent iterations.

[0059] Step S103: Determine the data processing result of the data to be processed based on the output matrix of the first model.

[0060] In this embodiment of the application, the output matrix of the first model can be determined as the data processing result of the data to be processed.

[0061] In this embodiment, during data processing, the data to be processed and a sparsity threshold set of the data processing model can be obtained first. The sparsity threshold set includes the sparsity threshold of each data processing module in the data processing model at each time step corresponding to each iteration. By pre-setting different sparsity thresholds for each data processing module at each time step, the sparsity of the data processing model can be improved. Then, the data to be processed is input into the data processing model, which performs multiple iterations to obtain a first model output matrix. In at least one iteration, a mask matrix is ​​determined based on the first activation feature vector output by the activation layer and the sparsity threshold. This mask matrix is ​​used for sparse computation in the current iteration and subsequent iterations. By determining the mask matrix, computation of the sparse portion can be skipped, reducing unnecessary calculations and improving model efficiency. Furthermore, the same mask matrix can be shared across multiple iterations, significantly reducing computational resource consumption in the mask matrix determination process. This ensures data processing accuracy while improving computational resource utilization. Finally, the data processing result is determined based on the output matrix of the first model. This improves both the sparsity of the data processing model and the computational efficiency of the model by using a mask matrix to achieve sparse computation.

[0062] The following examples illustrate the application scenarios of the data processing method provided in the embodiments of this application.

[0063] In the field of image generation, diffusion models can generate corresponding images based on text descriptions. To further improve the efficiency of the model, sparsification can be applied using a mask matrix. This sparsification method can significantly improve the inference speed of the diffusion model, making it more efficient in practical applications. For example, a designer creates promotional posters for a brand. He can input a simple text description. The sparsified diffusion model can quickly generate a series of high-quality images based on the received text description.

[0064] The data processing method of this application embodiment will be described below in conjunction with the above scenario.

[0065] Figure 2 This is another optional flowchart illustrating the data processing method provided in the embodiments of this application, such as... Figure 2 As shown, the method includes the following steps S201 to S210:

[0066] Step S201: The terminal receives data processing operations input by the user.

[0067] Data processing operations include selection operations or input operations, whereby the selection operation is used to select the data to be processed, or the input operation is used to input the data identifier of the data to be processed.

[0068] In step S202, the terminal encapsulates the data identifier of the data to be processed into the data processing request.

[0069] A data processing request is used to request the server to process data.

[0070] In step S203, the terminal sends a data processing request to the server.

[0071] In this embodiment of the application, the terminal sends a data processing request to the server to request the server to process the data.

[0072] In step S204, the server responds to the data processing request by obtaining the data to be processed and the sparsity threshold set of the data processing model.

[0073] In some embodiments, see Figure 3 , Figure 3 The step S204, "obtaining the sparsity threshold set of the data processing model," can be achieved through the following steps S2041 to S2045:

[0074] Step S2041: Input the second calibration data into the data processing model, and perform N rounds of iterative processing on the second calibration data through the data processing model to obtain the second model output matrix.

[0075] In this embodiment, the second calibration data can be obtained by randomly sampling from a real calibration dataset. The second calibration data can be data of any form: images, text, video, and audio, etc. It can be determined according to the specific task implemented by the data processing model. The second calibration data is input into the data processing model, which performs N rounds of iterative processing on the second calibration data. After N rounds of iterative processing, the second model output matrix is ​​obtained. For example, the data processing model includes two data processing modules (e.g., module A and module B). Modules A and B do not perform sparsity processing. The second calibration data is processed through modules A and B through N rounds of iterative processing to obtain the second model output matrix.

[0076] Step S2042: Obtain the candidate sparsity set.

[0077] In this embodiment of the application, the candidate sparsity set is preset, for example, the candidate sparsity set is (0.1, 0.2, 0.3).

[0078] Step S2043: For the Kth iteration, perform sparse processing on each data processing module in the data processing model according to each candidate sparsity in the candidate sparsity set, to obtain multiple sparse models corresponding to each candidate sparsity.

[0079] Here, K is an integer greater than 0 and less than or equal to N.

[0080] In this embodiment, K is an integer greater than 0 and less than or equal to N. That is, for each iteration in the N iterations, sparsity processing can be performed on one data processing module of the data processing model according to each candidate sparsity in the candidate sparsity set, thereby obtaining multiple sparse models corresponding to each candidate sparsity. For example, the candidate sparsity set is (0.1, 0.2, 0.3), and the data processing model includes two data processing modules (e.g., module A and module B). In each iteration, sparsity processing can be performed on each data processing module for each candidate sparsity. For example, if a total of 3 iterations are performed, the second iteration is used as an example. In the second iteration, when the candidate sparsity is 0.1, sparse processing can be performed on module A with a sparsity of 0.1, while module B is not subjected to sparsity processing. Furthermore, in the first and third iterations, neither module A nor module B is subjected to sparsity processing, resulting in a sparse model. Alternatively, sparse processing can be applied to module B with a sparsity of 0.1, while module A is not subjected to sparsity processing. Furthermore, in the first and third iterations, neither module A nor module B is subjected to sparsity processing, resulting in a sparse model. In the second iteration, when the candidate sparsity is 0.2, module A can be sparsified with a sparsity of 0.2, while module B is not sparsified. Furthermore, in the first and third iterations, neither module A nor module B is sparsified, resulting in a sparse model. Alternatively, module B can be sparsified with a sparsity of 0.2, while module A is not sparsified. Furthermore, in the first and third iterations, neither module A nor module B is sparsified, resulting in a sparse model. In the second iteration, when the candidate sparsity is 0.3, module A can be sparsified with a sparsity of 0.3, while module B is not sparsified. Furthermore, in the first and third iterations, neither module A nor module B is sparsified, resulting in a sparse model. Alternatively, module B can be sparsified with a sparsity of 0.3, while module A is not sparsified. Furthermore, in the first and third iterations, neither module A nor module B is sparsified, resulting in a sparse model. The implementation methods for other rounds of iteration are the same as those described above, and will not be repeated here. Ultimately, 18 sparse models can be obtained.

[0081] Step S2044: Input the second calibration data into each sparse model to obtain multiple third model output matrices.

[0082] In this embodiment, the second calibration data is iterated through N rounds using each sparse model to obtain the corresponding third model output matrix. For example, there are 18 sparse models, and the second calibration data is iterated through N rounds using each of the 18 sparse models to obtain the corresponding 18 third model output matrices.

[0083] Step S2045: Based on multiple third model output matrices and second model output matrices, determine the sparsity threshold set of the data processing model from the candidate sparsity set.

[0084] In some embodiments, step S2045 may also be implemented by performing the following processes: First, determining the cosine similarity between each third model output matrix and the second model output matrix; then, based on the cosine similarity, determining the target sparsity of each data processing module in the Kth round of iteration from the candidate sparsity set; finally, storing the target sparsity as the sparsity threshold of the data processing module in the Kth round of iteration in the sparsity threshold set.

[0085] In this embodiment, the cosine similarity between each third model output matrix and the second model output matrix can be calculated. Cosine similarity is an indicator that measures the degree of similarity between the third and second model output matrices, with values ​​ranging from -1 to 1. A higher cosine similarity indicates a closer similarity between the third and second model output matrices.

[0086] For each data processing module in the Kth iteration, in response to determining at least one target cosine similarity greater than a preset similarity threshold from all cosine similarities corresponding to candidate sparsities in the candidate sparsity set, the largest candidate sparsity is determined as the target sparsity of the data processing module in the Kth iteration. In response to not determining a target cosine similarity greater than the preset similarity threshold from all cosine similarities corresponding to candidate sparsities in the candidate sparsity set, the target sparsity of the data processing module in the Kth iteration is set to a preset value.

[0087] The similarity threshold can be preset, and can be preset according to the final output effect of the data processing model. This application embodiment does not limit this. For each data processing module in each round of iteration, the cosine similarity corresponding to each candidate sparsity is compared with the preset similarity threshold. When there is a cosine similarity greater than the preset similarity threshold, the candidate sparsity corresponding to the cosine similarity is determined as the target sparsity of the data processing module in this round of iteration. When there are multiple cosine similarities greater than the preset similarity threshold, the largest candidate sparsity among the multiple candidate sparsities corresponding to the multiple cosine similarities is determined as the target sparsity of the data processing module in this round of iteration. When there is no cosine similarity greater than the preset similarity threshold, the preset value is determined as the target sparsity of the data processing module in this round of iteration. The preset value can be set to 0, indicating that the data processing module does not perform sparsity processing in this round of iteration. The preset value can also be set to any number between 0 and 1 according to the actual situation. This application embodiment does not limit this. The following is an example. For example, a total of 3 rounds of iteration are performed. The second round of iteration is used as an example. Module A has three sparse models during the second round of iteration (sparse model sparse model sparse model Each sparse model corresponds to a third model output matrix. The cosine similarity between each third model output matrix and the second model output matrix is ​​calculated. For example, the obtained cosine similarities are for the sparse models... -0.5, sparse model -0.7, sparse model -0.9. If the preset similarity threshold is 0.8, then the target sparsity of module A in the second iteration is 0.9. If the preset similarity threshold is 0.6, then the target sparsity of module A in the second iteration is 0.9 (0.9 > 0.7). If the preset similarity threshold is 0.95, then the target sparsity of module A in the second iteration is 0 (i.e., the preset value is 0).

[0088] The target sparsity is used as the sparsity threshold for the data processing module in the Kth iteration and stored in the sparsity threshold set. For example, if the target sparsity of module A is 0.9 in the 2nd iteration, 0.9 can be used as the sparsity threshold for module A in the 2nd iteration and stored in the sparsity threshold set.

[0089] Through the above processing, the cosine similarity between the third model output matrix and the second model output matrix of each data processing module can be calculated in each iteration, dynamically determining the target sparsity for each module. When a cosine similarity greater than a preset similarity threshold exists, the largest candidate sparsity is selected as the target sparsity, thereby maximizing sparsity and minimizing computational resource consumption while ensuring the output performance of the data processing model. If no cosine similarity satisfies the condition, the target sparsity is set to a preset value to ensure the stability and effectiveness of the iteration process. Finally, the determined target sparsity is stored as a sparsity threshold in a sparsity threshold set, providing a reference for subsequent iterations and achieving a balance between sparsity processing and output performance in the data processing model.

[0090] Through steps S2041 to S2045, the second calibration data is input into the data processing model for multiple rounds of iterative processing to obtain the second model output matrix. Then, for each round of iterative processing, sparsity processing is performed on each data processing module in the data processing model according to each candidate sparsity in the candidate sparsity set, generating multiple sparse models. Next, the second calibration data is input into each sparse model to obtain multiple third model output matrices. Finally, based on the cosine similarity between the third model output matrix and the second model output matrix, a sparsity threshold set for the data processing model is determined from the candidate sparsity set. This allows for dynamic adjustment of sparsity, effectively reducing computational resource consumption and improving the operating efficiency of the data processing model while ensuring the output effect of the data processing model.

[0091] Step S205: The server obtains the set of channel importance scores for the data processing model.

[0092] Here, the channel importance score set includes the channel importance score of each channel in the activation layer.

[0093] In some embodiments, see Figure 4 , Figure 4 Step S205 can be achieved through the following steps S2051 to S2054:

[0094] Step S2051: Obtain the first calibration data.

[0095] In this embodiment, the first calibration data can be obtained by randomly sampling from a real calibration dataset. The first calibration data can be data of any form: images, text, video, and audio, etc. It can be determined according to the specific task implemented by the data processing model.

[0096] Step S2052: Input the first calibration data into the data processing model, and perform linear processing and activation processing on the first calibration data in sequence through the first linear layer and activation layer of the data processing model to obtain the second activation feature vector.

[0097] In this embodiment, the first calibration data is input into the data processing model. The first calibration data is first transformed linearly through a first linear layer to obtain an intermediate feature vector. Then, the intermediate feature vector is activated by an activation layer (such as ReLU, Sigmoid, etc.) to obtain a second activated feature vector.

[0098] Step S2053: Determine the absolute average value of the element corresponding to each channel of the activation layer in the second activation feature vector.

[0099] In this embodiment, for each channel of the activation layer, the average of the absolute values ​​of all elements in the second activation feature vector of that channel is calculated. Assuming the activation layer has C channels, the absolute average can be obtained using formula (1):

[0100] (1)

[0101] in, It is the absolute average. This is the total number of channels. It is a passage The absolute value of the element in the middle. This indicates calculating the average value.

[0102] Step S2054: Determine the absolute average value of each channel as the channel importance score, and obtain the channel importance score set.

[0103] In this embodiment, the absolute average value of each channel can be determined as the channel importance score. These channel importance scores are then aggregated to obtain a set of channel importance scores. That is, the absolute average value can be used as the channel importance score. As a score indicating the importance of the channel.

[0104] By using steps S2051 to S2054, the absolute average value of each channel is calculated as the channel importance score, which can quickly assess the importance of each channel and provide a reference for optimizing the data processing model.

[0105] In step S206, the server rearranges the channels of the activation layer in the data processing model based on the channel importance score of each channel to obtain the model parameters after channel rearrangement.

[0106] Here, the channel importance score set includes the channel importance score of each channel in the activation layer.

[0107] In some embodiments, see Figure 5 , Figure 5 Step S206 can be achieved through the following steps S2061 to S2063:

[0108] Step S2061: For each activation layer in the data processing model, obtain the weight parameters of each channel in the activation layer.

[0109] In this embodiment, the weight parameters of each channel in each activation layer of the data processing model are obtained. These weight parameters are learned parameters of the data processing model and are used to process the data to be processed during the forward propagation process. Each activation layer in the data processing model can be traversed. For each activation layer, the weight parameters of each channel in that layer are extracted. These weight parameters are stored in the parameter matrix of the data processing model and can be obtained by accessing the parameter matrix. For example, if an activation layer has 3 channels and the weight parameters are... , and .

[0110] Step S2062: Sort the second activation feature vectors according to the channel importance scores from smallest to largest to obtain the activation value channel rearranged vector.

[0111] In this embodiment, the channel feature vectors in the second activation feature vector are sorted according to the channel importance scores from smallest to largest to obtain the sorted second activation feature vector. The second activation feature vector can be mapped to the sorted second activation feature vector, which is the activation value channel rearrangement vector.

[0112] For example, the activation layer has three channels, denoted as C1, C2, and C3. The channel importance scores for each channel are 0.6, 0.4, and 0.8, respectively. The channels are sorted according to their importance scores from smallest to largest, resulting in the channel order C1, C2, and C3. Mapping the second activation feature vector to the sorted second activation feature vector yields the activation value channel rearrangement vector. This mapping vector can be... .

[0113] Step S2063: The weight parameters of each channel in the activation layer are transformed using the activation value channel rearrangement vector to obtain the model parameters after channel rearrangement.

[0114] In this embodiment, the weight parameters of each channel in the activation layer are transformed using the activation value channel rearrangement vector to obtain the model parameters after channel rearrangement. For example, the weight parameters are... , and The activation value channel rearrangement vector is... The model parameters after channel rearrangement are: , and .

[0115] Through steps S2061 to S2063, the weight parameters of each channel in each activation layer of the data processing model can be obtained, and the channel feature vectors are sorted according to the channel importance score to generate an activation value channel rearrangement vector. Then, the weight parameters are rearranged using this rearrangement vector to obtain the model parameters after channel rearrangement. This optimizes the structure of the data processing model, making the parameters of important channels in the data processing model more concentrated. This helps to maintain the performance and accuracy of the data processing model when it is subsequently sparsified, thereby reducing the consumption of computing resources while ensuring the output effect of the data processing model.

[0116] In step S207, the server inputs the data to be processed into the data processing model, and the data processing model performs multiple rounds of iterative processing on the data to be processed to obtain the first model output matrix.

[0117] Here, during at least one round of iteration, a mask matrix is ​​determined based on the first activation feature vector output by the activation layer and the sparsity threshold. The mask matrix is ​​used for sparse computation during the current iteration and subsequent iterations.

[0118] In some embodiments, step S207 can also be implemented by performing the following process: inputting the data to be processed into the channel rearranged data processing model corresponding to the channel rearranged model parameters, and performing multiple rounds of iterative processing on the data to be processed through the channel rearranged data processing model to obtain the first model output matrix.

[0119] In this embodiment, in step S206, after the server obtains the channel rearranged model parameters, the server can update the model parameters of the data processing model according to the channel rearranged model parameters to obtain the channel rearranged data processing model. Then, the data to be processed is input into the channel rearranged data processing model corresponding to the channel rearranged model parameters, and the data to be processed is subjected to multiple rounds of iterative processing through the channel rearranged data processing model to obtain the first model output matrix.

[0120] Through the above processing, channel rearrangement of model parameters can be completed offline, improving the running efficiency of the data processing model during online inference. The data to be processed is input into the channel-rearranged data processing model corresponding to the channel-rearranged model parameters, and multiple rounds of iterative processing are performed to obtain the first model output matrix. By utilizing the optimized channel-rearranged model parameters, the data processing model is more efficient in processing data, while maintaining the performance and accuracy of the data processing model.

[0121] In some embodiments, the above-mentioned "determining the mask matrix based on the first activation feature vector output by the activation layer and the sparsity threshold during at least one round of iteration processing" can also be achieved by performing the following processing: in response to determining that the current step is an iteration update step based on the iteration step number of the current step, obtaining the first activation feature vector output by the activation layer in the current step and the sparsity threshold of the data processing module where the activation layer is located in the current step; dividing the first activation feature vector into blocks according to a preset block size to obtain multiple sub-blocks; determining the importance score of each sub-block based on the elements in each sub-block; in response to any sub-block's importance score being greater than or equal to the sparsity threshold, determining the mask value corresponding to the sub-block in the mask matrix as the first mask value; in response to any sub-block's importance score being less than the sparsity threshold, determining the mask value corresponding to the sub-block in the mask matrix as the second mask value.

[0122] In this embodiment, the rules for iterative updates can be preset according to the number of iteration steps. For example, the first step can be set as the iterative update step, and then every certain number of steps, it can be set as the iterative update step again, such as every 3 steps, i.e., the first step, the fourth step, the seventh step, and so on, can be iterative update steps. When the current step is the iterative update step, the first activation feature vector output by the activation layer in the current step and the sparsity threshold of the data processing module where the activation layer in the current step is located can be obtained. Then, according to the preset block size, the first activation feature vector is divided into blocks to obtain multiple sub-blocks. For example, if the size of the first activation feature vector is M×N and the preset block size is m×n, the first activation feature vector is divided into blocks to obtain M / m×N / n sub-blocks. After that, the importance score of each sub-block can be calculated according to the preset rules for calculating the importance score of each sub-block. For example, the preset rules for calculating the importance score of each sub-block can be to calculate the absolute average of all elements in the sub-block, and then normalize the absolute average to obtain the normalized result. The normalized result is used as the importance score for that sub-block. Then, the importance score of each sub-block is compared with the sparsity threshold of the data processing module containing the activation layer. When the importance score of a sub-block is greater than or equal to the sparsity threshold, the mask value corresponding to that sub-block in the mask matrix is ​​determined as the first mask value, which can be set to 1. When the importance score of a sub-block is less than the sparsity threshold, the mask value corresponding to that sub-block in the mask matrix is ​​determined as the second mask value, which can be set to 0.

[0123] Through the above processing, in the iterative update step, by obtaining the first activation feature vector output by the activation layer in the current step and the corresponding sparsity threshold, the first activation feature vector is divided into blocks, the importance score of each sub-block is calculated, and the importance score is compared with the sparsity threshold, thereby determining the mask value of each sub-block in the mask matrix. This helps to dynamically adjust the sparsity of the data processing model during the inference process, optimize the structure of the data processing model, and improve the running efficiency and performance of the data processing model. It can reduce the consumption of computing resources while ensuring the output effect of the data processing model, and achieve efficient operation of the data processing model.

[0124] In some embodiments, the above-mentioned "determining the mask matrix" can also be implemented by performing the following process: in response to determining that the current step is a non-iterative update step based on the number of iteration steps of the current step, obtaining the mask matrix determined by the most recent iteration update step before the current step; and determining the mask matrix determined by the most recent iteration update step before the current step as the mask matrix of the current step.

[0125] In this embodiment of the application, if the current step is a non-iterative update step, the mask matrix determined by the most recent iterative update step before the current step can be obtained, and the mask matrix determined by the most recent iterative update step before the current step can be used as the mask matrix of the current step. For example, if steps 1, 4, 7, ... are iterative update steps, and the current step is step 6, the mask matrix determined by step 4 can be used as the mask matrix of the current step.

[0126] Through the above processing, in non-iterative update steps, the mask matrix determined in the most recent iteration update step can be used as the mask matrix for the current step. This effectively reduces computational load and improves the training efficiency of the data processing model. It reduces mask computation while maintaining the dynamic adjustment of the data processing model's sparsity, ensuring that the model maintains a reasonable sparse structure even in non-update steps. Therefore, while ensuring the performance of the data processing model, it significantly reduces computational resource consumption and improves the model's running speed and overall performance.

[0127] In some embodiments, the step S207 of "performing multiple rounds of iterative processing on the data to be processed through the data processing model to obtain the first model output matrix" can also be implemented by performing the following processing: First, in each round of iterative processing, for each data processing module in the data processing model, based on the mask matrix of the data processing module, the input features are processed by the first linear layer of the data processing module to obtain the first linear feature vector; wherein, for the first data processing module in the first round of iterative processing, the input feature is the data to be processed; for the first data processing module in the i-th round of iterative processing, the input feature is the model output matrix in the (i-1)-th round of iterative processing; for the j-th data processing module in the i-th round of iterative processing... The data processing module takes as input the output feature vector of the (j-1)th data processing module during the i-th iteration; where i is an integer greater than 1 and less than or equal to N; j is an integer greater than 1 and less than or equal to L, where L is the total number of data processing modules in the data processing model; then, the first linear feature vector is activated through the activation layer of the data processing module to obtain the third activated feature vector; subsequently, based on the mask matrix of the data processing module, the third activated feature vector is processed by the second linear layer of the data processing module to obtain the output feature vector of the data processing module; finally, the model output matrix of the last iteration is determined as the first model output matrix obtained after multiple iterations.

[0128] In this embodiment, during each iteration, for each data processing module in the data processing model, the input features are first divided into blocks according to a preset block size to obtain multiple input blocks. The mask value of each input block is determined from the mask matrix of the data processing module. If the mask value of any input block is the second mask value, the weight calculation for the input block is skipped. If the mask value of any input block is the first mask value, the input blocks are weighted and summed based on their weight parameters to obtain a first linear feature vector. For example, if the size of the input features is M×N and the preset block size is m×n, dividing the input features into blocks of size m×n yields M / m×N / n input blocks. Each input block corresponds to an element in the mask matrix of the data processing module. When the mask value of an input block is the second mask value, the weight calculation for the input block can be skipped. When the mask value of an input block is the first mask value, the input blocks are weighted and summed based on their weight parameters to obtain the first linear feature vector. By segmenting the input features into blocks according to a preset block size and determining whether to calculate weights for each input block based on the mask values ​​in the mask matrix, efficient utilization of computing resources is achieved. While maintaining the performance of the data processing model, the computational complexity is significantly reduced and the running efficiency of the data processing model is improved.

[0129] The first linear feature vector is activated by the activation layer of the data processing module to obtain the third activated feature vector.

[0130] The third activation feature vector is segmented into multiple input blocks according to a preset block size. The mask value for each input block is determined from the mask matrix of the data processing module. If the mask value of any input block is the second mask value, the weight calculation for the input block is skipped. If the mask value of any input block is the first mask value, the input blocks are weighted and summed based on their weight parameters to obtain the second linear feature vector. For example, if the size of the third activation feature vector is M×N and the preset block size is m×n, segmenting the third activation feature vector into blocks of size m×n yields M / m×N / n input blocks. Each input block corresponds to an element in the mask matrix of the data processing module. When the mask value of an input block is the second mask value, the weight calculation for the input block can be skipped. When the mask value of an input block is the first mask value, the input blocks are weighted and summed based on their weight parameters to obtain the second linear feature vector. By dividing the third activation feature vector into blocks according to a preset block size and determining whether to perform weight calculations on each input block based on the mask values ​​in the mask matrix, efficient utilization of computing resources is achieved. While maintaining the performance of the data processing model, the computational complexity is significantly reduced and the running efficiency of the data processing model is improved.

[0131] It should be noted that for the first data processing module in the first iteration, the input feature is the data to be processed. For the first data processing module in the i-th iteration, the input feature is the model output matrix in the (i-1)-th iteration. For example, a total of 3 iterations are performed, and the data processing model includes module A (equivalent to the first data processing module) and module B. For the first iteration, the input feature of module A is the data to be processed. For the second iteration, the input feature of module A is the model output matrix in the first iteration. For the j-th data processing module in the i-th iteration, the input feature is the output feature vector of the (j-1)-th data processing module in the i-th iteration; where i is an integer greater than 1 and less than or equal to N; j is an integer greater than 1 and less than or equal to L, where L is the total number of data processing modules in the data processing model. For example, for the second iteration, the input feature of module B is the output feature vector of module A in the second iteration.

[0132] The model output matrix from the last iteration is determined as the first model output matrix obtained after multiple iterations.

[0133] Through the above processing, in each iteration, a mask matrix is ​​used to control the processing of input features by the first linear layer for each data processing module in the data processing model, generating a first linear feature vector. Subsequently, further processing by activation layers and second linear layers yields the module's output feature vector. The mask matrix allows for skipping of unimportant features, reducing wasted computational resources and improving the efficiency of the data processing model. Finally, the output matrix of the last iteration is determined as the first model's output matrix, achieving gradual optimization and processing of data across multiple iterations, thus improving the performance and efficiency of the data processing model.

[0134] In step S208, the server determines the data processing result of the data to be processed based on the output matrix of the first model.

[0135] In step S209, the server sends the data processing results to the terminal.

[0136] In step S210, the terminal displays the data processing results.

[0137] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0138] Diffusion models have achieved great success in video and image generation. However, due to the large number of parameters and inference steps involved, they still suffer from significant latency. For example, a common video generation model can take several minutes to generate a 5-second 720P video. The data processing method proposed in this application can improve the computational efficiency of diffusion models.

[0139] This application employs a heterogeneous sparsity allocation method, assigning different sparsities to asynchronous and different blocks, resulting in high accuracy and sparsity for the entire diffusion model. To ensure a high speedup ratio, this application uses block-based structured sparsity, while utilizing channel sorting technology to ensure both sparsity and accuracy. To reduce the overhead introduced by sparse prediction, this application reuses the mask matrix from previous steps, eliminating the prediction process, and skipping a certain proportion of the linear computation of the first and second layers of the feed-forward network (FFN).

[0140] Compared to large language models, one reason why diffusion models struggle to improve sparsity is that they consist of different Transformer blocks and iterative loops (time steps) during denoising. Firstly, within the same time step, different Transformer blocks have varying degrees of sensitivity to sparsity. For example, under the same preset sparsity, most blocks can maintain high accuracy (accuracy refers to whether the output of the diffusion model differs significantly between using sparsification and not using sparsification (a large difference indicates low accuracy)), while a small number of blocks have very poor accuracy; these are the sensitive blocks. Secondly, the sparsity sensitivity of the same Transformer block also differs at different time steps because they perform different denoising tasks at different times during the denoising process.

[0141] To ensure the quality of the generated diffusion model, this application proposes a heterogeneous sparse allocation strategy. This involves assigning different preset sparsities to different time steps and different Transformer blocks in the diffusion model (i.e., the data processing model), ensuring that all blocks achieve a precision threshold during sparse computation. First, under different sparsities, the precision distribution of all steps and Transformer blocks is calculated, which is the cosine similarity threshold between sparse and full-precision computation. Second, given a precision threshold (which can be adjusted based on the final output of the diffusion model), the sparsity of all Transformer blocks at all steps is obtained. See [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of the sparsity distribution of different time steps and blocks provided in the embodiments of this application. It can be seen that the sparsity distribution of different time steps and different blocks is calculated using the embodiments of this application.

[0142] The following example illustrates this: For instance, we presuppose a diffusion model M, a set of candidate sparsity S (e.g., 0.01, 0.1, ..., 0.99), and an accuracy threshold. (i.e., a preset similarity threshold). Find a model that maintains its diffusion model performance at or above the threshold. Given the given conditions, this represents the maximum sparsity achievable. First, initialize the accuracy tensor: 3D tensor The dimension is the number of candidate sparsity. Time steps The number of blocks (i.e., data processing modules) 3D tensor This is used to store the accuracy at different sparsity levels, time steps, and blocks. Then, iterate through each candidate sparsity, performing the following steps for each candidate sparsity: Iterate through each time step and block; for each time step t and each block b, calculate the cosine similarity between the output of the sparse model (i.e., multiple third-model output matrices) and the output of the dense model (i.e., the second-model output matrix). Cosine similarity measures the directional similarity between two vectors; here, it is used to evaluate the similarity between the outputs of the sparse and dense models. This process is repeated for different time steps and blocks, eventually ending the loop for time steps and blocks, and then ending the loop for candidate sparsity. Finally, calculate the optimal sparsity mapping; for each time step t and each block b, find the mapping that maximizes accuracy. Greater than or equal to the accuracy threshold The maximum sparsity. The maximum sparsity (i.e., the sparsity threshold) can be stored in the optimal sparsity map, and then the computed optimal sparsity map is returned.

[0143] To improve the speedup ratio, the embodiments of this application can perform the following processing for FFN layers using the SiLU activation function.

[0144] First, generate a block mask. For the feature tensor after SiLU activation, the feature tensor has a size of M×N. Divide the feature tensor into M / m×N / n sub-blocks according to a block size of m×n (i.e., a preset block size). The importance score for each sub-block is then calculated. Calculated using the following formula (2):

[0145] (2)

[0146] in, It is an importance score. This indicates calculating the average value. For sub-blocks The absolute value of the element in the middle. It is a feature tensor.

[0147] Subsequently, by setting the value to apply to all sub-blocks of the feature tensor X sparsity threshold (Time step t, transformer block b) Generate a binary mask matrix The rules are as follows (3):

[0148] (3)

[0149] in, It is a sub-block The mask value, It is a sub-block Importance score It is the sparsity threshold.

[0150] Then, in the sparse computation stage, the binary mask matrix Blocks marked with 1 (i.e., the first mask value) participate in matrix multiplication calculations, while blocks marked with 0 (i.e., the second mask value) are not calculated and the all-zero submatrix is ​​returned directly.

[0151] Block sparsity methods can make extensive use of the tensor cores in the graphics processing unit (GPU) for efficient algorithm execution, with m×n sub-blocks perfectly aligned with the tile computation of the tensor cores. Simultaneously, block-granularity memory access operations guarantee 128-byte aligned memory transfers, significantly improving the bandwidth utilization of high-bandwidth memory (HBM); and compared to unstructured sparsity, the more regular access patterns also significantly improve L2 cache hit rate, thereby maintaining high inference performance. See also Figure 7 , Figure 7 This is a schematic diagram of the activation value mask distribution under different sparse methods provided in the embodiments of this application. It can be seen that different sparse methods yield different mask distributions. Figure 7 Part (a) uses a standard sparsity method (block size 1) for sparsity. Figure 7 Part (b) uses a direct block sparsity method for sparsity. Figure 7 Part (c) uses a block sparsity plus channel sorting method for sparsity. Figure 7 Part (a) in the document may result in some unimportant features being retained. Figure 7 Part (b) can better utilize the correlation between features, but may lead to the accidental deletion of some important features. Figure 7 Part (c) can more effectively utilize the correlation between features and better preserve important features.

[0152] The aforementioned block sparsity method suffers from significant accuracy issues in diffusion models because calculating importance scores involves averaging absolute values. When important activation values ​​(larger values ​​in the feature tensor X) coexist with numerous smaller values ​​(smaller values ​​in the feature tensor X), the importance score of the sub-block falls below the sparsity threshold, causing it to be skipped in the calculation. This leads to the accidental deletion of important features and a decrease in the accuracy of the diffusion model. To address this issue, the channel sorting technique provided in this application re-aggregates channels, grouping smaller values ​​together. This prevents sub-blocks containing important activation values ​​from being classified as sparsely computeable during the block sparsity process, thus significantly reducing accuracy.

[0153] Specifically, this application provides a static channel ranking method that uses data calibration to obtain the importance of different channels. First, a batch of calibration data (i.e., the first calibration data) is acquired. This data can be obtained by randomly sampling from real data. For each channel c of the feature tensor X (i.e., the second activated feature vector) after SiLU activation, the absolute average of all corresponding values ​​within that single channel is used as the channel's importance score. As shown in formula (1) above, it will not be repeated here. Then, according to... Reorder the channels from smallest to largest.

[0154] Figure 7 Part (b) shows the distribution of activation values ​​after channel sorting. The smaller activation values ​​are clustered in a closer region, which is beneficial for subsequent sparse operations.

[0155] See Figure 8 , Figure 8 This is a schematic diagram of channel sorting provided in an embodiment of this application. The size of the feature tensor is Token × Chan_IN, where Token represents the number of samples and Chan_IN represents the number of input channels. The first linear layer 801 converts the feature tensor into the output feature tensor Y1 through the weight matrix W1. The second linear layer 802 converts the output feature tensor Y1 into the final output feature tensor Y2 through the weight matrix W2. In this embodiment of the application, the weight matrix W1 can be rearranged through channel sorting. The output feature tensor Y1 can also be rearranged by calculating the rearranged weight matrix W1. For the output feature tensor Y1, each column is a channel, and 1, 2, 7, and 9 are the channel importance scores. It can be seen that each channel has a channel importance score. The rearranged vector P (i.e., the activation value channel rearranged vector) is the mapping vector that maps 7, 9, 2, 1 to 1, 2, 7, 9.

[0156] However, directly introducing channel reordering operations for activation values ​​in the FFN layer of the Transformer block would bring a significant computational burden. The direct channel sorting process is shown in formula (4):

[0157] (4)

[0158] in, It is a rearrangement function. Indicates the use of rearranged vectors To reorder the matrix , It is obtained by sorting the feature tensor X (i.e., the second activation feature vector) from smallest to largest channel.

[0159] Channel sorting of activation values ​​in the first linear layer of an FFN introduces two high-overhead activation value channel rearrangement operations, severely impacting inference efficiency. To address this issue, embodiments of this application propose a method to eliminate channel sorting overhead during inference by incorporating it into the offline weight transformation stage. Figure 8 The diagram illustrates the channel ranking process based on weight transformation. Specifically, before performing model inference, the diffusion model is transformed as shown in equations (5)-(6):

[0160] (5)

[0161] (6)

[0162] After transformation, the forward calculation can be simplified to formula (7):

[0163] (7)

[0164] in, This is the output of the diffusion model. It is a feature tensor. It's a reordering of weights. It's a reordering of weights. It is a mapping vector. These are the original weights. These are the original weights. It is a rearrangement function.

[0165] It can be seen that, according to Reorder and... according to After reordering, perform matrix multiplication directly and compare it with the original calculation result. Consistent.

[0166] Sparse methods often require mask calculation and index construction, which incurs high preprocessing overhead. To reduce this overhead, this application's embodiments, based on in-depth analysis of the diffusion model, reveal that the diffusion model exhibits a highly consistent sparsity distribution pattern between adjacent time steps; for example, the sparse mask similarity between steps five and four is as high as 95%. See also Figure 9 , Figure 9 This is a schematic diagram illustrating the principle of straddle mask multiplexing provided in the embodiments of this application. The embodiments of this application propose the following... Figure 9 The step mask reuse strategy shown achieves low computational cost and low latency mask acquisition. The specific process is as follows:

[0167] First, at a specified time step (e.g., step zero), perform one inference operation for the first linear layer 901 and the activation operation for SiLu (activation layer 902), and calculate the importance score for each block. The corresponding sparse mask can be generated by referring to formula (8). When calculating the second linear layer 903, a sparse mask can be used. Sparse computation is performed. Then, at each subsequent time step... In this process, the reference importance score from the initial step can be directly reused. Then, based on the current Transformer block b at the current time step t, the sparsity threshold obtained offline in the previous steps is used. Generate the corresponding sparse mask. As shown in formula (8):

[0168] (8)

[0169] in, These are the mask values ​​for each block. These are the importance scores for each block. It is the sparsity threshold

[0170] Then, use the generated mask Sparse computation is performed on the two linear layers of the FFN:

[0171] For the first linear layer of an FFN, the mask reveals the sparsity of the output block, indicating that the corresponding input multiplication computations can be skipped. Specifically, for the condition that satisfies... Output block The calculations in formula (9) can be skipped directly:

[0172] (9)

[0173] in, It is an output block. It is a sub-block. These are the weight parameters corresponding to the first linear layer. It can be seen that skipping each zero block reduces [weight parameters]. Sub-floating-point calculations.

[0174] For the second linear layer of FFN, a mask marks the sparse input blocks, indicating that multiply-accumulate operations with the corresponding weights can be skipped. Specifically, for the condition that... input block Skip the calculations in formula (10):

[0175] (10)

[0176] in, The input channel dimension is the weight. This is the output result. It is an input block. These are the weight parameters corresponding to the second linear layer. Skipping each zero block also saves [weights / weights]. Sub-floating-point calculations.

[0177] In addition, the embodiments of this application use block-granular storage of sparse masks. Compared with element-level masks, the memory mask storage overhead can be reduced by 2-3 orders of magnitude, which greatly improves memory efficiency. At the same time, the present invention designs an adaptive mask cache refresh mechanism, which recalculates the mask every K steps to ensure that the reused mask keeps pace with feature changes and improves the accuracy brought by reuse.

[0178] The embodiments of this application include two stages: offline weight transformation and online sparse inference. The specific process is as follows:

[0179] In the offline phase, the first step is to perform offline data calibration. The feature tensor X obtained during inference can be used, based on the calibrated data (i.e., the second calibration data), to statistically determine the sparsity threshold under different time steps and transformer block conditions. and channel importance score Then, offline weighted channel reordering is performed, based on the obtained channel importance. By using formulas (5)-(6), the weights are... and Rearrange to get and .

[0180] During the online inference phase, for each time step t, we first determine whether the sparse mask needs to be refreshed at the current time step. When t=0 or t is an integer multiple of the mask refresh interval K (i.e., the current step is an iterative update step), we perform the complete FFN first linear layer activation calculation. The feature tensor is divided into m×n blocks, and the importance score of each block is calculated. The importance score cache backup is updated. If the current step does not require a refresh (i.e., the current step is a non-iterative update step), the previous importance score cache backup is reused directly. The sparsity threshold of the current transformer block at the current time step is used. Then, the corresponding sparse mask is generated using the above formula (8). In the mask Once determined, the sparse computation phase begins. First, the first linear layer of the sparsified FFN is computed: for each output block... ,like Then execute If the mask value is 0, skip the calculation of this block and... The output is set to zero. Then, the sparsified FFN second linear layer computation is performed: traversing each input block. ,like Then matrix multiplication is performed. Otherwise, skip the accumulation operation for that block. Finally, output... This serves as the result of sparse inference for that step size.

[0181] Through the above process, the embodiments of this application significantly reduce floating-point computation and memory access overhead while ensuring sparse inference accuracy, and greatly improve the computational efficiency of the diffusion model in the inference stage.

[0182] Based on the data processing method described in the above embodiments Figure 10 The diagram shows a structural block diagram of a data processing device provided in an embodiment of this application. The data processing device 100 can be a device in an electronic device (e.g., a server). The data processing device can be implemented in software, which can be software in the form of programs and plug-ins, including the following software modules: data acquisition module 101, data processing module 102, and data output module 103. These modules are logically related, and therefore can be arbitrarily combined or further split according to the functions they implement.

[0183] The data acquisition module 101 is used to acquire the data to be processed and a sparsity threshold set of the data processing model. The data processing model includes multiple data processing modules, each including a first linear layer, an activation layer, and a second linear layer. The sparsity threshold set includes the sparsity threshold of each data processing module in the data processing model at each time step corresponding to each iteration. The data processing module 102 is used to input the data to be processed into the data processing model and perform multiple iterations on the data to be processed through the data processing model to obtain a first model output matrix. During at least one iteration, a mask matrix is ​​determined based on the first activation feature vector output by the activation layer and the sparsity threshold. The mask matrix is ​​used for sparse computation during the current iteration and subsequent iterations. The data output module 103 is used to determine the data processing result of the data to be processed based on the first model output matrix.

[0184] In some embodiments, the data acquisition module 101 is further configured to acquire a set of channel importance scores for the data processing model; the set of channel importance scores includes the channel importance score of each channel in the activation layer; based on the channel importance score of each channel, the activation layer in the data processing model is rearranged to obtain the rearranged model parameters; the step of inputting the data to be processed into the data processing model and performing multiple rounds of iterative processing on the data to be processed through the data processing model to obtain a first model output matrix includes: inputting the data to be processed into the channel rearranged data processing model corresponding to the rearranged model parameters, and performing multiple rounds of iterative processing on the data to be processed through the channel rearranged data processing model to obtain a first model output matrix.

[0185] In some embodiments, the data acquisition module 101 is further configured to acquire first calibration data; input the first calibration data into the data processing model, and sequentially perform linear processing and activation processing on the first calibration data through the first linear layer and activation layer of the data processing model to obtain a second activation feature vector; determine the absolute average value of the element corresponding to each channel of the activation layer in the second activation feature vector; and determine the absolute average value of each channel as the channel importance score of the channel to obtain the channel importance score set.

[0186] In some embodiments, the data acquisition module 101 is further configured to acquire the weight parameters of each channel in each activation layer of the data processing model; sort the second activation feature vector according to the channel importance scores in ascending order to obtain the activation value channel rearrangement vector; and use the activation value channel rearrangement vector to perform weight transformation on the weight parameters of each channel in the activation layer to obtain the model parameters after channel rearrangement.

[0187] In some embodiments, the data acquisition module 101 is further configured to: input second calibration data into the data processing model; perform N rounds of iterative processing on the second calibration data through the data processing model to obtain a second model output matrix; acquire a candidate sparsity set; for the Kth round of iterative processing, perform sparsity processing on each data processing module in the data processing model according to each candidate sparsity in the candidate sparsity set to obtain multiple sparse models corresponding to each candidate sparsity; K is an integer greater than 0 and less than or equal to N; input the second calibration data into each sparse model to obtain multiple third model output matrices; and determine the sparsity threshold set of the data processing model from the candidate sparsity set based on the multiple third model output matrices and the second model output matrix.

[0188] In some embodiments, the data acquisition module 101 is further configured to determine the cosine similarity between each third model output matrix and the second model output matrix; based on the cosine similarity, determine the target sparsity of each data processing module in the Kth round of iteration from the candidate sparsity set; and store the target sparsity as the sparsity threshold of the data processing module in the Kth round of iteration in the sparsity threshold set.

[0189] In some embodiments, the data acquisition module 101 is further configured to, for each data processing module in the Kth iteration, in response to determining at least one target cosine similarity greater than a preset similarity threshold from all cosine similarities corresponding to candidate sparsities in the candidate sparsity set, determine the largest candidate sparsity from at least one candidate sparsity corresponding to the at least one target cosine similarity as the target sparsity of the data processing module in the Kth iteration; and in response to not determining a target cosine similarity greater than the preset similarity threshold from all cosine similarities corresponding to candidate sparsities in the candidate sparsity set, set the target sparsity of the data processing module in the Kth iteration to a preset value.

[0190] In some embodiments, the data processing module 102 is further configured to, in response to determining that the current step is an iterative update step based on the number of iteration steps of the current step, obtain the first activation feature vector output by the activation layer in the current step and the sparsity threshold of the data processing module where the activation layer is located in the current step; divide the first activation feature vector into multiple sub-blocks according to a preset block size; determine the importance score of each sub-block based on the elements in each sub-block; in response to any sub-block having an importance score greater than or equal to the sparsity threshold, determine the mask value corresponding to the sub-block in the mask matrix as a first mask value; and in response to any sub-block having an importance score less than the sparsity threshold, determine the mask value corresponding to the sub-block in the mask matrix as a second mask value.

[0191] In some embodiments, the data processing module 102 is further configured to, in response to determining that the current step is a non-iterative update step based on the number of iteration steps of the current step, obtain the mask matrix determined by the most recent iteration update step before the current step; and determine the mask matrix determined by the most recent iteration update step before the current step as the mask matrix of the current step.

[0192] In some embodiments, the data processing module 102 is further configured to, in each iteration, perform a first linear processing on the input features based on the mask matrix of the data processing module, through the first linear layer of the data processing module, to obtain a first linear feature vector for each data processing module in the data processing model; wherein, for the first data processing module in the first iteration, the input feature is the data to be processed; for the first data processing module in the i-th iteration, the input feature is the model output matrix in the (i-1)-th iteration; and for the j-th data processing module in the i-th iteration, the input feature is the model output matrix in the i-th iteration. The output feature vector of the (j-1)th data processing module; where i is an integer greater than 1 and less than or equal to N; j is an integer greater than 1 and less than or equal to L, where L is the total number of data processing modules in the data processing model; the first linear feature vector is activated by the activation layer of the data processing module to obtain the third activated feature vector; based on the mask matrix of the data processing module, the third activated feature vector is processed by the second linear layer of the data processing module to obtain the output feature vector of the data processing module; the model output matrix of the last iteration is determined as the first model output matrix obtained after the multi-round iteration.

[0193] In some embodiments, the data processing module 102 is further configured to: divide the input features into blocks according to a preset block size to obtain multiple input blocks; determine the mask value of each input block from the mask matrix of the data processing module; skip the weight calculation of the input block in response to the mask value of any input block being a second mask value; and perform a weighted summation of the input blocks based on the weight parameters of the input blocks in response to the mask value of any input block being a first mask value to obtain the first linear feature vector.

[0194] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0195] This application provides an electronic device. Figure 11 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 11 As shown, the electronic device 130 includes: at least one processor 131 ( Figure 11 (Only one is shown in the diagram), memory 132, and computer-executable instructions 133 stored in memory 132 and executable on at least one processor 131, wherein processor 131 executes computer-executable instructions 133 to implement the steps in any of the above-described data processing method embodiments.

[0196] The electronic device may include, but is not limited to, a processor 131 and a memory 132. Those skilled in the art will understand that... Figure 11 This is merely an example of electronic device 130 and does not constitute a limitation on electronic device 130. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0197] Processor 131 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0198] In some embodiments, memory 132 may be an internal storage unit of electronic device 130, such as a hard disk or memory of electronic device 130. In other embodiments, memory 132 may be an external storage device of electronic device 130, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on electronic device 130. Furthermore, memory 132 may include both internal and external storage units of electronic device 130. Memory 132 is used to store operating system, application programs, bootloader, data, and other programs, such as program code of computer programs. Memory 132 may also be used to temporarily store data that has been output or will be output.

[0199] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data processing method described in this application.

[0200] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the data processing method provided in this application. For example, ... Figure 1 The data processing method is shown.

[0201] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0202] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0203] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0204] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0205] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Receive data processing requests sent by the terminal; In response to the data processing request, the system acquires the data to be processed and inputs the second calibration data into the data processing model. The data processing model then performs N rounds of iterative processing on the second calibration data to obtain a second model output matrix. A candidate sparsity set is acquired. For the Kth round of iterative processing, sparse processing is performed on each data processing module in the data processing model according to each candidate sparsity in the candidate sparsity set, resulting in multiple sparse models corresponding to each candidate sparsity; K is an integer greater than 0 and less than or equal to N. The second calibration data is input into each sparse model to obtain multiple third model output matrices. Based on the... The data processing model is determined from the candidate sparsity set by using multiple third model output matrices and the second model output matrix. The data processing model includes multiple data processing modules, each including a first linear layer, an activation layer, and a second linear layer. The sparsity threshold set includes the sparsity threshold of each data processing module in the data processing model at each time step corresponding to each iteration. The data to be processed is any one of image data, text data, video data, and audio data. The data processing model is used to implement an image generation task or a video generation task. The data to be processed is input into the data processing model, and the data to be processed is subjected to multiple rounds of iterative processing through the data processing model to obtain the first model output matrix; wherein, in at least one round of iterative processing, a mask matrix is ​​determined based on the first activation feature vector output by the activation layer and the sparsity threshold, and the mask matrix is ​​used to perform sparse calculation in the current step of iterative processing and in multiple steps of iterative processing after the current step; The data processing result of the data to be processed is determined based on the output matrix of the first model. The data processing result is an image generated by the corresponding image generation task or a video generated by the corresponding video generation task.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the channel importance score set of the data processing model; the channel importance score set includes the channel importance score of each channel in the activation layer; Based on the channel importance score of each channel, the activation layer in the data processing model is rearranged to obtain the model parameters after channel rearrangement. The step of inputting the data to be processed into the data processing model, and performing multiple rounds of iterative processing on the data to be processed through the data processing model to obtain a first model output matrix includes: The data to be processed is input into the channel rearrangement data processing model corresponding to the channel rearrangement model parameters. The data to be processed is then processed through multiple rounds of iterative processing by the channel rearrangement data processing model to obtain the first model output matrix.

3. The method according to claim 2, characterized in that, The acquisition of the channel importance score set of the data processing model includes: Obtain the first calibration data; The first calibration data is input into the data processing model, and the first calibration data is linearly processed and activated sequentially through the first linear layer and the activation layer of the data processing model to obtain the second activation feature vector. Determine the absolute average value of the elements corresponding to each channel of the activation layer in the second activation feature vector; The absolute average value of each channel is determined as the channel importance score of that channel, thus obtaining the set of channel importance scores.

4. The method according to claim 3, characterized in that, The activation layer in the data processing model is reordered based on the channel importance score of each channel to obtain the reordered model parameters, including: For each activation layer in the data processing model, obtain the weight parameters of each channel in the activation layer; The second activation feature vector is sorted according to the channel importance scores from smallest to largest to obtain the activation value channel rearranged vector. The weight parameters of each channel in the activation layer are transformed using the channel rearrangement vector of the activation value to obtain the model parameters after channel rearrangement.

5. The method according to claim 1, characterized in that, The step of determining the sparsity threshold set of the data processing model from the candidate sparsity set based on the plurality of third model output matrices and the second model output matrix includes: Determine the cosine similarity between each third model output matrix and the second model output matrix; Based on the cosine similarity, the target sparsity of each data processing module in the Kth round of iteration is determined from the candidate sparsity set; The target sparsity is used as the sparsity threshold of the data processing module during the Kth iteration and stored in the sparsity threshold set.

6. The method according to claim 5, characterized in that, The step of determining the target sparsity of each data processing module in the Kth round of iteration based on the cosine similarity from the candidate sparsity set includes: For each data processing module in the Kth iteration, In response to determining at least one target cosine similarity greater than a preset similarity threshold from all cosine similarities corresponding to candidate sparsities in the candidate sparsity set, the maximum candidate sparsity is determined as the target sparsity of the data processing module in the Kth iteration process from at least one candidate sparsity corresponding to the at least one target cosine similarity. In response to the fact that no target cosine similarity greater than a preset similarity threshold is determined from all cosine similarities corresponding to candidate sparsities in the candidate sparsity set, the target sparsity of the data processing module in the Kth iteration is set to a preset value.

7. The method according to claim 1, characterized in that, The step of determining the mask matrix based on the first activation feature vector output by the activation layer and the sparsity threshold during at least one round of iteration includes: In response to determining that the current step is an iterative update step based on the number of iteration steps of the current step, the first activation feature vector output by the activation layer under the current step and the sparsity threshold of the data processing module where the activation layer is located under the current step are obtained. The first activation feature vector is divided into multiple sub-blocks according to a preset block size. Based on the elements in each sub-block, determine the importance score of each sub-block; In response to any sub-block having an importance score greater than or equal to the sparsity threshold, the mask value corresponding to the sub-block in the mask matrix is ​​determined as the first mask value; In response to any sub-block having an importance score less than the sparsity threshold, the mask value corresponding to the sub-block in the mask matrix is ​​determined as the second mask value.

8. The method according to claim 7, characterized in that, The method further includes: In response to determining that the current step is a non-iterative update step based on the number of iterations in the current step, the mask matrix determined by the most recent iteration update step before the current step is obtained; The mask matrix determined in the most recent iteration update step before the current step is used as the mask matrix for the current step.

9. The method according to any one of claims 1 to 8, characterized in that, The process of performing multiple rounds of iterative processing on the data to be processed using the data processing model to obtain the first model output matrix includes: In each round of iterative processing, for each data processing module in the data processing model, Based on the mask matrix of the data processing module, the input features are processed by the first linear layer of the data processing module to obtain a first linear feature vector; wherein, for the first data processing module in the first round of iteration, the input feature is the data to be processed; for the first data processing module in the i-th round of iteration, the input feature is the model output matrix in the (i-1)-th round of iteration; for the j-th data processing module in the i-th round of iteration, the input feature is the output feature vector of the (j-1)-th data processing module in the i-th round of iteration; wherein, i is an integer greater than 1 and less than or equal to N; j is an integer greater than 1 and less than or equal to L, and L is the total number of data processing modules in the data processing model; The first linear feature vector is activated by the activation layer of the data processing module to obtain the third activated feature vector; Based on the mask matrix of the data processing module, the third activation feature vector is subjected to second linear processing through the second linear layer of the data processing module to obtain the output feature vector of the data processing module; The model output matrix of the last iteration is determined as the first model output matrix obtained after the multiple iterations.

10. The method according to claim 9, characterized in that, The mask matrix based on the data processing module, through the first linear layer of the data processing module, performs a first linear processing on the input features to obtain a first linear feature vector, including: The input features are divided into multiple input blocks according to a preset block size. The mask value for each input block is determined from the mask matrix of the data processing module; In response to the mask value of any input block being the second mask value, the weight calculation for the input block is skipped; In response to the mask value of any input block being a first mask value, the input blocks are weighted and summed based on the weight parameters of the input blocks to obtain the first linear feature vector.

11. A data processing apparatus, characterized in that, The device includes: A data acquisition module is used to receive data processing requests sent by a terminal; in response to the data processing requests, acquire data to be processed, and input second calibration data into a data processing model, perform N rounds of iterative processing on the second calibration data through the data processing model to obtain a second model output matrix; acquire a candidate sparsity set; for the Kth round of iterative processing, perform sparse processing on each data processing module in the data processing model according to each candidate sparsity in the candidate sparsity set to obtain multiple sparse models corresponding to each candidate sparsity; K is an integer greater than 0 and less than or equal to N; input the second calibration data into each sparse model to obtain multiple The third model output matrix; based on the plurality of third model output matrices and the second model output matrix, a sparsity threshold set for the data processing model is determined from the candidate sparsity set; wherein, the data processing model includes a plurality of data processing modules, each data processing module including a first linear layer, an activation layer and a second linear layer; the sparsity threshold set includes: the sparsity threshold of each data processing module in the data processing model at the time step corresponding to each round of iteration processing; the data to be processed is any one of image data, text data, video data and audio data; the data processing model is used to implement an image generation task or a video generation task; The data processing module is used to input the data to be processed into the data processing model, and to perform multiple rounds of iterative processing on the data to be processed through the data processing model to obtain a first model output matrix; wherein, in at least one round of iterative processing, a mask matrix is ​​determined based on the first activation feature vector output by the activation layer and the sparsity threshold, and the mask matrix is ​​used to perform sparse calculation in the current step of iterative processing and in multiple steps of iterative processing after the current step; The data output module is used to determine the data processing result of the data to be processed based on the first model output matrix. The data processing result is an image generated by a corresponding image generation task or a video generated by a corresponding video generation task.

12. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the data processing method according to any one of claims 1 to 10.

14. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the data processing method according to any one of claims 1 to 10.

Images