Data processing method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a data processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring to-be-processed data and a sparseness threshold set of a data processing model; inputting the to-be-processed data into the data processing model, and performing multi-round iterative processing on the to-be-processed data through the data processing model to obtain a first model output matrix; wherein during at least one round of iterative processing, a mask matrix is determined based on a first activation feature vector output by the activation layer and a sparsity threshold value, and the mask matrix is used for performing sparse calculation during the current step of iterative processing and multiple steps of iterative processing after the current step; and determining a data processing result of the to-be-processed data based on the first model output matrix. According to the invention, the calculation efficiency of the data processing model can be improved.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a data processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] In deep learning models, as data volumes and model complexity continue to increase, the demand for computing resources also grows exponentially. This not only increases the time required for training and inference, but also places higher demands on hardware resources. Although sparsification can reduce unnecessary computations and parameters in a model, effectively reducing computational complexity and storage requirements, in practice, sparsification can lead to a decrease in model accuracy. While maintaining model accuracy, the model can only undergo limited sparsification, resulting in low computational efficiency. Summary of the Invention

[0003] The embodiments of the present application provide a data processing method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the computational efficiency of data processing models.

[0004] The technical solution of the embodiment of the present application is implemented as follows: An embodiment of the present application provides a data processing method, the method comprising: obtaining data to be processed and a sparsity threshold set of a data processing model; wherein the data processing model comprises a plurality of data processing modules, each of which comprises a first linear layer, an activation layer, and a second linear layer; the sparsity threshold set comprises: a sparsity threshold of each data processing module in the data processing model at a time step corresponding to each round of iterative processing; inputting the data to be processed into the data processing model, performing multiple rounds of iterative processing on the data to be processed through the data processing model to obtain a first model output matrix; wherein, in at least one round of iterative processing, a mask matrix is determined based on a first activation feature vector output by the activation layer and the sparsity threshold, the mask matrix being used to perform sparse calculations in a current iterative processing step and multiple iterative processing steps after the current step; and determining a data processing result of the data to be processed based on the first model output matrix.

[0005] An embodiment of the present application provides a data processing device, comprising: a data acquisition module, configured to acquire data to be processed and a sparsity threshold set of a data processing model; wherein the data processing model comprises a plurality of data processing modules, each of which comprises a first linear layer, an activation layer, and a second linear layer; the sparsity threshold set comprises: a sparsity threshold of each data processing module in the data processing model at a time step corresponding to each round of iterative processing; a data processing module, configured to input the data to be processed into the data processing model, and perform multiple rounds of iterative processing on the data to be processed through the data processing model to obtain a first model output matrix; wherein, during at least one round of iterative processing, a mask matrix is determined based on the first activation feature vector output by the activation layer and the sparsity threshold, and the mask matrix is used to perform sparse calculation during the current iterative processing and multiple iterative processing steps after the current step; and a data output module, configured to determine a data processing result of the data to be processed based on the first model output matrix.

[0006] In the above scheme, the data acquisition module is also used to obtain a channel importance score set of the data processing model; the channel importance score set includes the channel importance score of each channel in the activation layer; based on the channel importance score of each channel, the activation layer in the data processing model is channel-rearranged to obtain the model parameters after channel rearrangement; the data to be processed is input into the data processing model, and the data processing model is used to perform multiple rounds of iterative processing on the data to be processed to obtain the first model output matrix, including: inputting the data to be processed into the channel rearranged data processing model corresponding to the channel-rearranged model parameters, and the channel rearranged data processing model is used to perform multiple rounds of iterative processing on the data to be processed to obtain the first model output matrix.

[0007] In the above scheme, the data acquisition module is also used to obtain first calibration data; input the first calibration data into the data processing model, and perform linear processing and activation processing on the first calibration data in sequence through the first linear layer and activation layer of the data processing model to obtain a second activation feature vector; determine the absolute average value of the elements corresponding to each channel of the activation layer in the second activation feature vector; determine the absolute average value of each channel as the channel importance score of the channel, and obtain the channel importance score set.

[0008] In the above scheme, the data acquisition module is also used to obtain the weight parameters of each channel in each activation layer in the data processing model; sort the channel feature vectors of the second activation feature vector in order from small to large according to the channel importance score to obtain the activation value channel rearrangement vector; use the activation value channel rearrangement vector to perform weight transformation on the weight parameters of each channel in the activation layer to obtain the model parameters after channel rearrangement.

[0009] In the above scheme, the data acquisition module is also used to input the second calibration data into the data processing model, perform N rounds of iterative processing on the second calibration data through the data processing model to obtain a second model output matrix; obtain a candidate sparsity set; for the Kth round of iterative processing, perform sparse processing on each data processing module in the data processing model according to each candidate sparsity in the candidate sparsity set to obtain multiple sparse models corresponding to each candidate sparsity; K is an integer greater than 0 and less than or equal to N; input the second calibration data into each sparse model to obtain multiple third model output matrices; based on the multiple third model output matrices and the second model output matrix, determine the sparsity threshold set of the data processing model from the candidate sparsity set.

[0010] In the above scheme, the data acquisition module is also used to determine the cosine similarity between each third model output matrix and the second model output matrix; based on the cosine similarity, determine the target sparsity of each data processing module during the K-th round of iterative processing from the candidate sparsity set; use the target sparsity as the sparsity threshold of the data processing module during the K-th round of iterative processing, and store it in the sparsity threshold set.

[0011] In the above scheme, the data acquisition module is further used to, for each data processing module during the K-th round of iterative processing, in response to determining at least one target cosine similarity greater than a preset similarity threshold from all cosine similarities corresponding to the candidate sparsities in the candidate sparsity set, determine the maximum candidate sparsity as the target sparsity of the data processing module during the K-th round of iterative processing from at least one candidate sparsity corresponding to the at least one target cosine similarity; in response to not determining a target cosine similarity greater than the preset similarity threshold from all cosine similarities corresponding to the candidate sparsities in the candidate sparsity set, set the target sparsity of the data processing module during the K-th round of iterative processing to a preset value.

[0012] In the above scheme, the data processing module is also used to, in response to determining that the current step is an iterative update step based on the number of iterations of the current step, obtain the first activation feature vector output by the activation layer under the current step and the sparsity threshold of the data processing module where the activation layer is located under the current step; block the first activation feature vector according to a preset block size to obtain multiple sub-blocks; determine the importance score of each sub-block based on the elements in each sub-block; in response to the importance score of any sub-block being greater than or equal to the sparsity threshold, determine the mask value corresponding to the sub-block in the mask matrix as the first mask value; in response to the importance score of any sub-block being less than the sparsity threshold, determine the mask value corresponding to the sub-block in the mask matrix as the second mask value.

[0013] In the above scheme, the data processing module is further used to obtain the mask matrix determined by the most recent iterative update step before the current step in response to determining that the current step is a non-iterative update step based on the number of iterative steps of the current step; and determine the mask matrix determined by the most recent iterative update step before the current step as the mask matrix of the current step.

[0014] In the above scheme, the data processing module is further used to, in each round of iterative processing, for each data processing module in the data processing model, perform a first linear processing on the input feature through the first linear layer of the data processing module based on the mask matrix of the data processing module to obtain a first linear feature vector; wherein, for the first data processing module in the first round of iterative processing, the input feature is the data to be processed; for the first data processing module in the i-th round of iterative processing, the input feature is the model output matrix in the i-1-th round of iterative processing; for the j-th data processing module in the i-th round of iterative processing, the input feature is the model output matrix in the i-1-th round of iterative processing output eigenvectors of j-1 data processing modules; wherein i is an integer greater than 1 and less than or equal to N; j is an integer greater than 1 and less than or equal to L, and L is the total number of data processing modules in the data processing model; the first linear eigenvector is activated by the activation layer of the data processing module to obtain a third activated eigenvector; based on the mask matrix of the data processing module, the third activated eigenvector is subjected to a second linear processing by the second linear layer of the data processing module to obtain the output eigenvector of the data processing module; the model output matrix of the last round of iterative processing is determined as the first model output matrix obtained after the multiple rounds of iterative processing.

[0015] In the above scheme, the data processing module is also used to block the input features according to a preset block size to obtain multiple input blocks; determine the mask value of each input block from the mask matrix of the data processing module; in response to the mask value of any input block being the second mask value, skip the weight calculation of the input block; in response to the mask value of any input block being the first mask value, perform weighted summation on the input blocks based on the weight parameters of the input blocks to obtain the first linear feature vector.

[0016] An embodiment of the present application provides an electronic device, comprising: a memory for storing computer-executable instructions or computer programs; and a processor for implementing the data processing method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0017] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data processing method provided in the embodiment of the present application when executed by a processor.

[0018] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the data processing method provided in the embodiment of the present application is implemented.

[0019] The embodiments of the present application have the following beneficial effects: When performing data processing, a set of sparsity thresholds of the data to be processed and the data processing model can be first obtained, and the sparsity threshold set includes: the sparsity threshold of each data processing module in the data processing model at the time step corresponding to each round of iterative processing. By presetting different sparsity thresholds for each data processing module at each time step, the sparsity of the data processing model can be improved. Then, the data to be processed is input into the data processing model, and the data processing model performs multiple rounds of iterative processing on the data to be processed to obtain a first model output matrix; wherein, during at least one round of iterative processing, a mask matrix is determined based on the first activated eigenvector output by the activation layer and the sparsity threshold. The mask matrix is used to perform sparse calculations during the current iterative processing and multiple iterative processing steps after the current step. In this way, by determining the mask matrix, the calculation of the sparse part can be skipped, unnecessary operations can be reduced, and the operating efficiency of the model can be improved. The same mask matrix can be shared during multiple iterative processing steps, and the computing resource consumption of the mask matrix determination process can be greatly reduced. While ensuring data processing accuracy, the computing resource utilization rate can also be improved. Finally, the data processing result of the data to be processed is determined based on the first model output matrix. In this way, the sparsity of the data processing model is improved, and sparse calculation is achieved by using the mask matrix, which can improve the computational efficiency of the data processing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is an optional flowchart of the data processing method provided in the embodiment of the present application; Figure 2 This is another optional flowchart of the data processing method provided in the embodiment of the present application; Figure 3 This is a schematic diagram of the implementation process of obtaining the sparsity threshold provided in an embodiment of the present application; Figure 4 This is a schematic diagram of the implementation process of obtaining channel importance scores provided in an embodiment of the present application; Figure 5 1 is a schematic diagram of an implementation flow of obtaining model parameters after channel rearrangement provided in an embodiment of the present application; Figure 6 is a schematic diagram of the sparsity distribution of different time steps and blocks provided by an embodiment of the present application; Figure 7 Schematic diagram of activation value mask distribution under different sparse methods provided in the embodiments of the present application; Figure 8 is a schematic diagram of channel sorting provided in an embodiment of the present application; Figure 9 Schematic diagram of the principle of stride mask multiplexing provided by an embodiment of the present application; Figure 10 This is a structural block diagram of a data processing device provided in an embodiment of the present application; Figure 11 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0022] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0023] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0024] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or portion of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal. It can be implemented in whole or in part using software, hardware (such as processing circuits or memory), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the functionality of the module or unit.

[0025] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0026] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0027] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0028] 1) In response to: This field indicates the conditions or states upon which the executed operation depends. When the conditions or states are met, the executed operation or operations may be performed in real time or with a set delay. Unless otherwise specified, the executed operations may be executed in any order.

[0029] 2) Sparsity: Sparsity is a metric that measures the proportion of nonzero elements in data. It can be used to describe the proportion of zero elements in the weight matrix of a data processing module. A higher sparsity indicates a higher proportion of zero elements and a lower proportion of nonzero elements in the data processing module.

[0030] 3) Importance score: A value used to measure the importance of each sub-block. The larger the importance score, the more important the sub-block.

[0031] 4) Channel importance score: A value used to measure the importance of each channel. The larger the channel importance score, the more important the channel.

[0032] In order to better understand the data processing method provided in the embodiments of the present application, the data processing method in the related art is first described below.

[0033] In related technologies, to achieve contextual sparsity prediction, a sparsity predictor composed of a multi-layer perceptron is introduced. The sparsity predictor itself uses additional weight parameters and needs to be retrained for each target model, which brings about a large training overhead and reduces the applicability and reusability of the method. On the other hand, during the model inference process, the sparsity predictor needs to dynamically infer the input features in real time, resulting in additional computational burden and inference delay, thereby limiting the acceleration effect of sparse inference in actual end-to-end inference time and failing to fully unleash the performance potential of sparsity. Although structured pruning methods can be used to achieve low-cost sparsification without retraining, the pruning effect still has certain limitations. Specifically, the pruning ratio is limited, usually only achieving a sparsity rate of about 10%, and the pruning amplitude is low, resulting in an inability to achieve significant inference acceleration. Second, due to the insufficient sparsity rate, the number of parameters and computational complexity of the pruned model are still large, which cannot fully unleash the performance potential brought by model compression, limiting the application effect of the model in resource-constrained environments.

[0034] Based on the problems existing in the related technology, an embodiment of the present application provides a data processing method. When performing data processing, a set of sparsity thresholds of the data to be processed and the data processing model can be obtained first. The sparsity threshold set includes: the sparsity threshold of each data processing module in the data processing model at the time step corresponding to each round of iterative processing. By pre-setting different sparsity thresholds for each data processing module at each time step, the sparsity of the data processing model can be improved. Then, the data to be processed is input into the data processing model, and the data processing model is used to perform multiple rounds of iterative processing on the data to be processed to obtain a first model output matrix; wherein, during at least one round of iterative processing, a mask matrix is determined based on the first activated eigenvector output by the activation layer and a sparsity threshold value, and the mask matrix is used to perform sparse calculations during the current step iterative processing and the multiple-step iterative processing after the current step. In this way, by determining the mask matrix, the calculation of the sparse part can be skipped, unnecessary operations can be reduced, and the operating efficiency of the model can be improved. In addition, the same mask matrix can be shared during the multiple-step iterative processing, and the computing resource consumption of the mask matrix determination process can be greatly reduced. While ensuring data processing accuracy, the computing resource utilization rate can also be improved. Finally, the data processing result of the data to be processed is determined based on the first model output matrix. In this way, the sparsity of the data processing model is improved, and sparse calculation is achieved through the mask matrix, thereby improving the computing efficiency of the data processing model.

[0035] The data processing method provided in the embodiments of the present application can be applied to electronic devices such as laptop computers, tablet computers, and desktop computers. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0036] The data processing method provided in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0037] Figure 1 This is an optional flow chart of the data processing method provided by the embodiment of the present application, which can be applied to electronic devices. The following will be illustrated by taking the electronic device as a server as an example. It should be noted that the data processing method provided by the embodiment of the present application can at least be applied to the generation of large models based on diffusion, with the purpose of accelerating the reasoning efficiency of the generation of large models. Figure 1 As shown, the method includes the following steps S101 to S103: Step S101: Obtain the data to be processed and the sparsity threshold set of the data processing model.

[0038] Here, the data processing model includes multiple data processing modules, each of which includes a first linear layer, an activation layer, and a second linear layer. The sparsity threshold set includes: a sparsity threshold for each data processing module in the data processing model at a time step corresponding to each round of iterative processing.

[0039] In the embodiments of the present application, the data to be processed can be any form of data: images, text, video, and audio. This can be determined based on the specific tasks implemented by the data processing model. The specific tasks implemented by the data processing model can be video generation tasks or image generation tasks. For example, the task can be a text-to-video task or a picture-to-video task within a video generation task, or the task can be a text-to-picture task or a picture-to-picture task within an image generation task. When the task is a text-to-video task, the data to be processed can at least be text data, and based on the input text data, the data processing method provided in the embodiments of the present application can be used to generate videos of different styles and types. When the task is a picture-to-video task, the data to be processed can at least be image data, and based on the input image data, the data processing method provided in the embodiments of the present application can be used to generate videos of different styles and types. When the task is a text-to-picture task, the data to be processed can at least be text data, and based on the input text data, the data processing method provided in the embodiments of the present application can be used to generate images of different styles and types. When the task is a picture-to-picture task, the data to be processed can at least be image data, and based on the input image data, the data processing method provided in the embodiments of the present application can be used to generate new images.

[0040] The data processing model can be a Transformer model, a convolutional neural network, a multimodal fusion model, etc. This embodiment of the present application does not limit this, and can be selected according to actual circumstances. The data processing model can include multiple data processing modules. Each data processing module can include a first linear layer, an activation layer, and a second linear layer. For example, if the data processing model is a Transformer model, the Transformer model can include multiple Transformer modules. Each Transformer module can include a first linear layer, an activation layer, and a second linear layer.

[0041] Sparsity is an indicator that measures the proportion of non-zero elements in data. Sparsity can be used to describe the proportion of zero elements in the weight matrix of a data processing module. The higher the sparsity, the higher the proportion of zero elements and the lower the proportion of non-zero elements in the data processing module. The sparsity threshold set includes: the sparsity threshold of each data processing module in the data processing model at the time step corresponding to each round of iterative processing, that is, each data processing module has a sparsity threshold at different time steps. For a time step, the sparsity threshold of each data processing module can be the same or different. For the same data processing module, the sparsity threshold of the data processing module can be the same or different at different time steps.

[0042] Step S102: input the data to be processed into the data processing model, and perform multiple rounds of iterative processing on the data to be processed through the data processing model to obtain a first model output matrix.

[0043] Here, in at least one round of iterative processing, a mask matrix is determined based on the first activated feature vector output by the activation layer and the sparsity threshold. The mask matrix is used to perform sparse calculations in the current step iterative processing and multiple steps of iterative processing after the current step.

[0044] In embodiments of the present application, multiple rounds of iterative processing may refer to the process by which a data processing model processes the data to be processed multiple times. Each round of processing is called an iteration. The first model output matrix may refer to the final output matrix generated by the data processing model after multiple rounds of iterative processing. This matrix may contain features or results processed by the data processing model. For example, if the data to be processed is image data, the first model output matrix may be a probability distribution representing the probability that the input image belongs to different categories. An activation layer refers to a nonlinear layer introduced in the data processing model. Common activation functions include ReLU, Sigmoid, Tanh, and GELU. The first activated feature vector refers to the feature vector output by the activation layer. For example, the feature vector processed by the ReLU activation function in the feedforward network layer of a Transformer model. The mask matrix may be a matrix of the same shape as the first activated feature vector, used to indicate whether the vector at each position in the first activated feature vector can be skipped (i.e., sparse calculation). The values in the mask matrix are typically 0 or 1, with 0 indicating that the vector at that position can be skipped and 1 indicating that the vector at that position needs to be calculated.

[0045] During multiple rounds of iterative processing of the data to be processed by the data processing model, a mask matrix can be determined based on the first activated eigenvector output by the activation layer and a sparsity threshold during at least one round of iterative processing. The mask matrix is used to perform sparse calculations during the current iterative processing step and multiple iterative processing steps following the current step.

[0046] Step S103: determining a data processing result of the data to be processed based on the first model output matrix.

[0047] In the embodiment of the present application, the first model output matrix can be determined as the data processing result of the data to be processed.

[0048] In the embodiment of the present application, when performing data processing, a set of sparsity threshold values of the data to be processed and the data processing model can be first obtained, and the sparsity threshold value set includes: the sparsity threshold value of each data processing module in the data processing model at the time step corresponding to each round of iterative processing. By presetting different sparsity threshold values for each data processing module at each time step, the sparsity of the data processing model can be improved. Then, the data to be processed is input into the data processing model, and the data processing model performs multiple rounds of iterative processing on the data to be processed to obtain a first model output matrix; wherein, in at least one round of iterative processing, a mask matrix is determined based on the first activated eigenvector output by the activation layer and the sparsity threshold value, and the mask matrix is used to perform sparse calculations during the current step iterative processing and the multiple steps after the current step iterative processing. In this way, by determining the mask matrix, the calculation of the sparse part can be skipped, unnecessary operations can be reduced, and the operating efficiency of the model can be improved. In addition, the same mask matrix can be shared during the multiple steps of iterative processing, and the computing resource consumption of the mask matrix determination process can be greatly reduced. While ensuring the data processing accuracy, the computing resource utilization rate can also be improved. Finally, the data processing result of the data to be processed is determined based on the first model output matrix. In this way, the sparsity of the data processing model is improved, and sparse calculation is achieved through the mask matrix, thereby improving the computational efficiency of the data processing model.

[0049] The following examples illustrate the application scenarios of the data processing method provided in the embodiments of the present application.

[0050] In the field of image generation, diffusion models can generate images based on text descriptions. To further improve model efficiency, diffusion models can be sparsified using a mask matrix. This sparsification method significantly increases the inference speed of diffusion models, making them more efficient in practical applications. For example, a designer designing a promotional poster for a brand can input a simple text description. The sparsified diffusion model can then quickly generate a series of high-quality images based on the received text description.

[0051] The data processing method of the embodiment of the present application will be described below in combination with the above scenario.

[0052] Figure 2 This is another optional flow chart of the data processing method provided in the embodiment of the present application, such as Figure 2As shown, the method includes the following steps S201 to S210: Step S201: The terminal receives a data processing operation input by a user.

[0053] The data processing operation includes a selection operation or an input operation. The selection operation is used to select data to be processed, or the input operation is used to input a data identifier of the data to be processed.

[0054] In step S202 , the terminal encapsulates the data identifier of the data to be processed into a data processing request.

[0055] Data processing requests are used to request the server to process data.

[0056] Step S203: The terminal sends a data processing request to the server.

[0057] In an embodiment of the present application, the terminal sends a data processing request to the server to request the server to process the data.

[0058] Step S204: The server obtains the data to be processed and the sparsity threshold set of the data processing model in response to the data processing request.

[0059] In some embodiments, see Figure 3 , Figure 3 The step S204 of “obtaining a set of sparsity threshold values of a data processing model” can be implemented by the following steps S2041 to S2045: Step S2041: input the second calibration data into the data processing model, and perform N rounds of iterative processing on the second calibration data through the data processing model to obtain a second model output matrix.

[0060] In an embodiment of the present application, the second calibration data can be obtained by randomly sampling from a real calibration data set. The second calibration data can be data in any form: image, text, video, audio, etc. It can be determined according to the specific task implemented by the data processing model. The second calibration data is input into the data processing model, and the second calibration data can be iteratively processed N times by the data processing model. After N times of iterative processing, the second model output matrix is obtained. For example, the data processing model includes two data processing modules (for example, module A and module B), and module A and module B are not sparsely processed. The second calibration data is iteratively processed N times by module A and module B to obtain the second model output matrix.

[0061] Step S2042: Obtain a candidate sparsity set.

[0062] In the embodiment of the present application, the candidate sparsity set is pre-set, for example, the candidate sparsity set is (0.1, 0.2, 0.3).

[0063] Step S2043: for the Kth round of iterative processing, sparse processing is performed on each data processing module in the data processing model according to each candidate sparsity in the candidate sparsity set to obtain multiple sparse models corresponding to each candidate sparsity.

[0064] Here, K is an integer greater than 0 and less than or equal to N.

[0065] In an embodiment of the present application, K is an integer greater than 0 and less than or equal to N, that is, for each round of iterative processing in N rounds of iterative processing, a data processing module in the data processing model can be sparsely processed in turn according to each candidate sparsity in the candidate sparsity set, thereby obtaining multiple sparse models corresponding to each candidate sparsity. For example, the candidate sparsity set is (0.1, 0.2, 0.3), and the data processing model includes two data processing modules (for example, module A and module B). In each round of iterative processing, the data processing modules can be sparsely processed for each candidate sparsity. For example, a total of 3 rounds of iterative processing are performed, and the second round of iterative processing is used as an example. In the second round of iterative processing, when the candidate sparsity is 0.1, module A can be sparsely processed according to the sparsity of 0.1, and module B is not sparsely processed. Moreover, in the first and third rounds of iterative processing, neither module A nor module B is sparsely processed, and a sparse model is obtained. ; Module B can also be sparsely processed according to the sparsity of 0.1, and module A is not sparsely processed. In addition, during the first and third rounds of iterative processing, neither module A nor module B is sparsely processed, and a sparse model is obtained. In the second round of iterative processing, when the candidate sparsity is 0.2, module A can be sparsely processed according to the sparsity of 0.2, and module B is not sparsely processed. In addition, in the first and third rounds of iterative processing, neither module A nor module B is sparsely processed, and a sparse model is obtained. ; Module B can also be sparsely processed according to the sparsity of 0.2, and module A is not sparsely processed. In addition, during the first and third rounds of iterative processing, neither module A nor module B is sparsely processed, and a sparse model is obtained. In the second round of iterative processing, when the candidate sparsity is 0.3, module A can be sparsely processed according to the sparsity of 0.3, and module B is not sparsely processed. In addition, in the first and third rounds of iterative processing, neither module A nor module B is sparsely processed, and a sparse model is obtained. ; Module B can also be sparsely processed according to the sparsity of 0.3, and module A is not sparsely processed. In addition, during the first and third rounds of iterative processing, neither module A nor module B is sparsely processed, and a sparse model is obtained. The implementation of other rounds of iterative processing is the same as the above implementation, which will not be repeated here. Finally, 18 sparse models can be obtained.

[0066] Step S2044: input the second calibration data into each sparse model to obtain multiple third model output matrices.

[0067] In the embodiment of the present application, each sparse model performs N rounds of iterative processing on the second calibration data to obtain a corresponding third model output matrix. For example, there are 18 sparse models in total, and each of the 18 sparse models performs N rounds of iterative processing on the second calibration data to obtain 18 corresponding third model output matrices.

[0068] Step S2045: Determine a sparsity threshold set of the data processing model from the candidate sparsity set based on the multiple third model output matrices and the second model output matrix.

[0069] In some embodiments, step S2045 can also be implemented by performing the following processing: first, determine the cosine similarity between each third model output matrix and the second model output matrix; then, based on the cosine similarity, determine the target sparsity of each data processing module during the Kth round of iterative processing from the candidate sparsity set; finally, use the target sparsity as the sparsity threshold of the data processing module during the Kth round of iterative processing and store it in the sparsity threshold set.

[0070] In an embodiment of the present application, the cosine similarity between each third model output matrix and the second model output matrix can be calculated. The cosine similarity can be an indicator of the similarity between the third model output matrix and the second model output matrix, and the cosine similarity value ranges from -1 to 1. The larger the cosine similarity, the closer the third model output matrix is to the second model output matrix.

[0071] For each data processing module during the Kth round of iterative processing, in response to determining at least one target cosine similarity greater than a preset similarity threshold from all cosine similarities corresponding to candidate sparsities in the candidate sparsity set, a maximum candidate sparsity is determined as the target sparsity of the data processing module during the Kth round of iterative processing from at least one candidate sparsity corresponding to the at least one target cosine similarity. In response to not determining a target cosine similarity greater than the preset similarity threshold from all cosine similarities corresponding to the candidate sparsities in the candidate sparsity set, the target sparsity of the data processing module during the Kth round of iterative processing is set to a preset value.

[0072] The similarity threshold can be preset and can be preset according to the final output effect of the data processing model, which is not limited in the embodiments of the present application. For each data processing module during each round of iterative processing, the cosine similarity corresponding to each candidate sparsity is compared with the preset similarity threshold. When there is a cosine similarity greater than the preset similarity threshold, the candidate sparsity corresponding to the cosine similarity is determined as the target sparsity of the data processing module during the round of iterative processing. When there are multiple cosine similarities greater than the preset similarity threshold, the largest candidate sparsity among the multiple candidate sparsities corresponding to the multiple cosine similarities is determined as the target sparsity of the data processing module during the round of iterative processing. When no cosine similarity is greater than the preset similarity threshold, the preset value is determined as the target sparsity of the data processing module during the round of iterative processing. The preset value can be set to 0, indicating that the data processing module does not perform sparse processing during the round of iterative processing. The preset value can also be set to any number between 0 and 1 according to actual conditions, which is not limited in the embodiments of the present application. The following example illustrates, for example, a total of 3 rounds of iterative processing are performed, and the second round of iterative processing is used as an example. When module A is iterated in the second round, there are three sparse models (sparse models , sparse model , sparse model ). Each sparse model corresponds to a third model output matrix. Calculate the cosine similarity between each third model output matrix and the second model output matrix. For example, the cosine similarities obtained are sparse models -0.5, sparse model -0.7, sparse model -0.9. If the preset similarity threshold is 0.8, the target sparsity of module A in the second round of iterative processing is 0.9. If the preset similarity threshold is 0.6, the target sparsity of module A in the second round of iterative processing is 0.9 (0.9>0.7). If the preset similarity threshold is 0.95, the target sparsity of module A in the second round of iterative processing is 0 (that is, the preset value is 0).

[0073] The target sparsity is used as the sparsity threshold for the data processing module during the Kth round of iterative processing and stored in the sparsity threshold set. For example, if the target sparsity of module A during the second round of iterative processing is 0.9, 0.9 can be used as the sparsity threshold for module A during the second round of iterative processing and stored in the sparsity threshold set.

[0074] Through the above processing, the cosine similarity between the third model output matrix and the second model output matrix of each data processing module can be calculated in each round of iterative processing, and the target sparsity of each module can be dynamically determined. When there is a cosine similarity greater than the preset similarity threshold, the maximum candidate sparsity is selected as the target sparsity, so as to improve the sparsity as much as possible and reduce the consumption of computing resources while ensuring the output effect of the data processing model. If there is no cosine similarity that meets the conditions, the target sparsity is set to a preset value to ensure the stability and effectiveness of the iterative process. Finally, the determined target sparsity is stored as a sparsity threshold in the sparsity threshold set to provide a reference for subsequent iterative processing, thereby achieving a balanced optimization of the data processing model between sparse processing and output effect.

[0075] Through steps S2041 to S2045, the second calibration data can be input into the data processing model for multiple rounds of iterative processing to obtain a second model output matrix. Then, for each round of iterative processing, each data processing module in the data processing model is sparsely processed according to each candidate sparsity in the candidate sparsity set to generate multiple sparse models. Next, the second calibration data is input into each sparse model to obtain multiple third model output matrices. Finally, based on the cosine similarity between the third model output matrix and the second model output matrix, the sparsity threshold set of the data processing model is determined from the candidate sparsity set, and the sparsity can be dynamically adjusted. While ensuring the output effect of the data processing model, it is possible to effectively reduce the consumption of computing resources and improve the operating efficiency of the data processing model.

[0076] Step S205: The server obtains a channel importance score set of the data processing model.

[0077] Here, the channel importance score set includes the channel importance score of each channel in the activation layer.

[0078] In some embodiments, see Figure 4 , Figure 4 It is shown that step S205 can be implemented by the following steps S2051 to S2054: Step S2051, obtaining first calibration data.

[0079] In the embodiment of the present application, the first calibration data can be randomly sampled from a real calibration data set. The first calibration data can be data in any form: image, text, video, audio, etc. It can be determined based on the specific task to be implemented by the data processing model.

[0080] Step S2052: input the first calibration data into the data processing model, and sequentially perform linear processing and activation processing on the first calibration data through the first linear layer and activation layer of the data processing model to obtain a second activated feature vector.

[0081] In this embodiment of the present application, the first calibration data is input into the data processing model. The first calibration data is first linearly transformed by a first linear layer to obtain an intermediate feature vector. The intermediate feature vector is then activated by an activation layer (such as ReLU or Sigmoid) to obtain a second activated feature vector.

[0082] Step S2053: Determine the absolute average value of the elements corresponding to each channel of the activation layer in the second activation feature vector.

[0083] In this embodiment of the present application, for each channel of the activation layer, the average of the absolute values of all elements in the second activation feature vector of the channel is calculated. Assuming that the activation layer has C channels, the absolute average value can be obtained by formula (1): (1) in, is the absolute mean, is the total number of channels, It is a channel The absolute value of the elements in , Indicates finding the average value.

[0084] Step S2054 : Determine the absolute average value of each channel as the channel importance score of the channel, and obtain a channel importance score set.

[0085] In the embodiment of the present application, the absolute average value of each channel can be determined as the channel importance score of the channel, and these channel importance scores are aggregated to obtain a channel importance score set. as the channel importance score.

[0086] Through steps S2051 to S2054, by calculating the absolute average value of each channel as the channel importance score, the importance of each channel can be quickly evaluated, providing a reference basis for the optimization of the data processing model.

[0087] In step S206 , the server performs channel rearrangement on the activation layer in the data processing model based on the channel importance score of each channel to obtain model parameters after channel rearrangement.

[0088] Here, the channel importance score set includes the channel importance score of each channel in the activation layer.

[0089] In some embodiments, see Figure 5 , Figure 5 It shows that step S206 can be implemented by the following steps S2061 to S2063: Step S2061: For each activation layer in the data processing model, obtain the weight parameters of each channel in the activation layer.

[0090] In the embodiment of the present application, the weight parameters of each channel of each activation layer in the data processing model are obtained. These weight parameters are the parameters learned by the data processing model and are used to process the data to be processed during the forward propagation process. Each activation layer in the data processing model can be traversed. For each activation layer, the weight parameters of each channel in the layer are extracted. These weight parameters are stored in the parameter matrix of the data processing model and can be obtained by accessing the parameter matrix. For example, there are 3 channels in the activation layer and the weight parameters are 、 and .

[0091] Step S2062 , sorting the second activation feature vectors by channel feature vectors in ascending order of channel importance scores to obtain an activation value channel rearrangement vector.

[0092] In an embodiment of the present application, the channel feature vectors in the second activation feature vector are sorted in ascending order according to the channel importance scores to obtain a sorted second activation feature vector. The second activation feature vector can be mapped to a mapping vector of the sorted second activation feature vector, that is, an activation value channel rearrangement vector.

[0093] For example, the activation layer has three channels, denoted as C1, C2, and C3. The channel importance scores of each channel are 0.6, 0.4, and 0.8 respectively. The channels are sorted in ascending order according to the channel importance scores, and the sorted channel order is C1, C2, and C3. The second activation feature vector is mapped to the mapping vector of the sorted second activation feature vector, which is the activation value channel rearrangement vector. The mapping vector can be .

[0094] Step S2063: Use the activation value channel rearrangement vector to perform weight transformation on the weight parameters of each channel in the activation layer to obtain the model parameters after channel rearrangement.

[0095] In the embodiment of the present application, the activation value channel rearrangement vector is used to perform weight transformation on the weight parameters of each channel in the activation layer to obtain the model parameters after channel rearrangement. For example, the weight parameter is 、 and The activation value channel rearrangement vector is The model parameters after channel rearrangement are 、 and .

[0096] Through steps S2061 to S2063, the weight parameters of each channel of each activation layer in the data processing model can be obtained, and the channel feature vectors are sorted according to the channel importance scores to generate an activation value channel rearrangement vector. Then, the weight parameters are rearranged using this rearrangement vector to obtain the model parameters after channel rearrangement, which optimizes the structure of the data processing model and makes the parameters of important channels in the data processing model more concentrated. This helps to maintain the performance and accuracy of the data processing model when the data processing model is subsequently sparsely processed, thereby reducing computing resource consumption while ensuring the output effect of the data processing model.

[0097] In step S207 , the server inputs the data to be processed into the data processing model, and performs multiple rounds of iterative processing on the data to be processed through the data processing model to obtain a first model output matrix.

[0098] Here, in at least one round of iterative processing, a mask matrix is determined based on the first activated feature vector output by the activation layer and the sparsity threshold. The mask matrix is used to perform sparse calculations in the current step iterative processing and multiple steps of iterative processing after the current step.

[0099] In some embodiments, step S207 can also be implemented by performing the following processing: inputting the data to be processed into the channel rearrangement data processing model corresponding to the model parameters after channel rearrangement, performing multiple rounds of iterative processing on the data to be processed through the channel rearrangement data processing model to obtain the first model output matrix.

[0100] In this embodiment of the present application, in step S206, after the server obtains the model parameters after the channel rearrangement, the server can update the model parameters of the data processing model according to the channel rearrangement model parameters to obtain a channel rearrangement data processing model. The data to be processed is then input into the channel rearrangement data processing model corresponding to the channel rearrangement model parameters. The channel rearrangement data processing model performs multiple rounds of iterative processing on the data to obtain a first model output matrix.

[0101] Through this process, the channel rearrangement of model parameters can be completed offline, improving the operational efficiency of the data processing model during online inference. The data to be processed is input into the channel rearrangement data processing model corresponding to the channel rearrangement model parameters, and multiple rounds of iterative processing are performed to obtain the first model output matrix. Utilizing the optimized channel rearrangement model parameters, the data processing model is made more efficient when processing data while maintaining its performance and accuracy.

[0102] In some embodiments, the above-mentioned "determining the mask matrix based on the first activation feature vector output by the activation layer and the sparsity threshold in at least one round of iterative processing" can also be achieved by performing the following processing: in response to determining that the current step is an iterative update step based on the number of iterations of the current step, obtaining the first activation feature vector output by the activation layer under the current step and the sparsity threshold of the data processing module where the activation layer under the current step is located; according to a preset block size, the first activation feature vector is divided into blocks to obtain multiple sub-blocks; based on the elements in each sub-block, the importance score of each sub-block is determined; in response to the importance score of any sub-block being greater than or equal to the sparsity threshold, the mask value corresponding to the sub-block in the mask matrix is determined as the first mask value; in response to the importance score of any sub-block being less than the sparsity threshold, the mask value corresponding to the sub-block in the mask matrix is determined as the second mask value.

[0103] In an embodiment of the present application, an iterative update rule can be pre-set based on the number of iteration steps. For example, the first step is set as the iterative update step, and then every certain number of steps is set as the iterative update step again. For example, every three steps is set as the iterative update step again, that is, the first, fourth, seventh, and so on are iterative update steps. When the current step is the iterative update step, the first activation feature vector output by the activation layer at the current step and the sparsity threshold of the data processing module where the activation layer at the current step is located can be obtained. Then, the first activation feature vector is block-partitioned according to a preset block size to obtain multiple sub-blocks. For example, if the size of the first activation feature vector is M×N and the preset block size is m×n, the first activation feature vector is block-partitioned to obtain M / m×N / n sub-blocks. Thereafter, the importance score of each sub-block can be calculated according to a preset rule for calculating the importance score of each sub-block. For example, the preset rule for calculating the importance score of each sub-block can be to calculate the absolute average of all elements in the sub-block and then normalize the absolute average to obtain a normalized result. The normalized result is used as the importance score of the sub-block. Afterwards, the importance score of each sub-block is compared with the sparsity threshold of the data processing module where the activation layer is located. When the importance score of the sub-block is greater than or equal to the sparsity threshold, the mask value corresponding to the sub-block in the mask matrix is determined as the first mask value, and the first mask value can be set to 1. When the importance score of the sub-block is less than the sparsity threshold, the mask value corresponding to the sub-block in the mask matrix can be determined as the second mask value, and the second mask value can be set to 0.

[0104] Through the above processing, in the iterative update step, by obtaining the first activated eigenvector output by the activation layer under the current step and the corresponding sparsity threshold, the first activated eigenvector can be segmented into blocks, the importance score of each sub-block can be calculated, and the importance score can be compared with the sparsity threshold to determine the mask value of each sub-block in the mask matrix. This helps to dynamically adjust the sparsity of the data processing model during the data processing model inference process, optimize the structure of the data processing model, and improve the operating efficiency and performance of the data processing model. It can reduce the consumption of computing resources while ensuring the output effect of the data processing model, and realize the efficient operation of the data processing model.

[0105] In some embodiments, the above-mentioned "determining the mask matrix" can also be achieved by performing the following processing: in response to determining that the current step is a non-iterative update step based on the number of iterations of the current step, obtaining the mask matrix determined in the most recent iterative update step before the current step; and determining the mask matrix determined in the most recent iterative update step before the current step as the mask matrix of the current step.

[0106] In an embodiment of the present application, if the current step is a non-iterative update step, the mask matrix determined in the most recent iterative update step before the current step can be obtained and used as the mask matrix for the current step. For example, if steps 1, 4, 7, ... are iterative update steps and the current step is step 6, the mask matrix determined in step 4 can be used as the mask matrix for the current step.

[0107] Through this process, during non-iterative update steps, the mask matrix determined in the most recent iterative update step can be used as the mask matrix for the current step, effectively reducing computational effort and improving data processing model training efficiency. This reduces mask calculations while maintaining dynamic adjustment of the data processing model's sparsity, ensuring that the model maintains a reasonable sparse structure even during non-update steps. This significantly reduces computational resource consumption while ensuring the model's performance, improving its speed and overall performance.

[0108] In some embodiments, the step S207 of "performing multiple rounds of iterative processing on the data to be processed by the data processing model to obtain a first model output matrix" can also be implemented by performing the following processing: first, in each round of iterative processing, for each data processing module in the data processing model, based on the mask matrix of the data processing module, the first linear layer of the data processing module performs a first linear processing on the input feature to obtain a first linear feature vector; wherein, for the first data processing module in the first round of iterative processing, the input feature is the data to be processed; for the first data processing module in the i-th round of iterative processing, the input feature is the model output matrix in the i-1-th round of iterative processing; for the j-th data processing module in the i-th round of iterative processing, the input feature is the model output matrix in the i-1-th round of iterative processing; The data processing module, the input feature is the output feature vector of the j-1th data processing module in the i-th round of iterative processing; wherein i is an integer greater than 1 and less than or equal to N; j is an integer greater than 1 and less than or equal to L, and L is the total number of data processing modules in the data processing model; then, the first linear feature vector is activated by the activation layer of the data processing module to obtain a third activated feature vector; thereafter, based on the mask matrix of the data processing module, the third activated feature vector is subjected to a second linear processing by the second linear layer of the data processing module to obtain the output feature vector of the data processing module; finally, the model output matrix of the last round of iterative processing is determined to be the first model output matrix obtained after multiple rounds of iterative processing.

[0109] In an embodiment of the present application, during each round of iterative processing, for each data processing module in the data processing model, the input features are first block-segmented according to a preset block size to obtain multiple input blocks. A mask value for each input block is determined from the mask matrix of the data processing module. If the mask value of any input block is the second mask value, weight calculation for the input block is skipped. If the mask value of any input block is the first mask value, a weighted summation of the input blocks is performed based on the weight parameters of the input blocks to obtain a first linear feature vector. For example, if the input feature size is M×N and the preset block size is m×n, the input features are block-segmented according to the block size of m×n to obtain M / m×N / n input blocks. Each input block corresponds to an element in the mask matrix of the data processing module. When the mask value of the input block is the second mask value, weight calculation for the input block can be skipped. When the mask value of the input block is the first mask value, a weighted summation of the input blocks is performed based on the weight parameters of the input blocks to obtain the first linear feature vector. By dividing the input features into blocks according to the preset block size and deciding whether to perform weight calculation on each input block based on the mask value in the mask matrix, efficient utilization of computing resources is achieved. While maintaining the performance of the data processing model, the computational complexity is significantly reduced and the operating efficiency of the data processing model is improved.

[0110] The first linear eigenvector is activated by an activation layer of a data processing module to obtain a third activated eigenvector.

[0111] The third activated eigenvector is block-segmented according to a preset block size to obtain multiple input blocks; a mask value for each input block is determined from a mask matrix of a data processing module; in response to the mask value of any input block being the second mask value, weight calculation for the input block is skipped; in response to the mask value of any input block being the first mask value, a weighted summation of the input blocks is performed based on the weight parameters of the input blocks to obtain a second linear eigenvector. For example, if the size of the third activated eigenvector is M×N and the preset block size is m×n, the third activated eigenvector is block-segmented according to the block size of m×n to obtain M / m×N / n input blocks. Each input block corresponds to an element in the mask matrix of the data processing module. When the mask value of the input block is the second mask value, weight calculation for the input block can be skipped. When the mask value of the input block is the first mask value, a weighted summation of the input blocks can be performed based on the weight parameters of the input blocks to obtain a second linear eigenvector. By dividing the third activated eigenvector into blocks according to a preset block size and deciding whether to perform weight calculation on each input block based on the mask value in the mask matrix, efficient utilization of computing resources is achieved. While maintaining the performance of the data processing model, the computational complexity is significantly reduced, and the operating efficiency of the data processing model is improved.

[0112] It should be noted that for the first data processing module in the first round of iterative processing, the input features are the data to be processed. For the first data processing module in the i-th round of iterative processing, the input features are the model output matrix in the i-1th round of iterative processing. For example, a total of three rounds of iterative processing are performed, and the data processing model includes module A (equivalent to the first data processing module) and module B. For the first round of iterative processing, the input features of module A are the data to be processed. For the second round of iterative processing, the input features of module A are the model output matrix in the first round of iterative processing. For the j-th data processing module in the i-th round of iterative processing, the input features are the output feature vectors of the j-1-th data processing module in the i-th round of iterative processing; where i is an integer greater than 1 and less than or equal to N; j is an integer greater than 1 and less than or equal to L, and L is the total number of data processing modules in the data processing model. For example, for the second round of iterative processing, the input features of module B are the output feature vectors of module A in the second round of iterative processing.

[0113] The model output matrix of the last round of iterative processing is determined as the first model output matrix obtained after multiple rounds of iterative processing.

[0114] Through the above process, in each round of iterative processing, for each data processing module in the data processing model, a mask matrix can be used to control the processing of input features by the first linear layer, generating a first linear feature vector. Subsequently, further processing is performed through the activation layer and the second linear layer to obtain the module's output feature vector. The mask matrix is used to skip unimportant features, reducing the waste of computing resources and improving the operational efficiency of the data processing model. Ultimately, the output matrix of the last iteration is determined as the first model output matrix, achieving gradual optimization and processing of data over multiple rounds of iterations, thereby improving the performance and efficiency of the data processing model.

[0115] Step S208: The server determines a data processing result of the data to be processed based on the first model output matrix.

[0116] Step S209: The server sends the data processing result to the terminal.

[0117] Step S210: The terminal displays the data processing result.

[0118] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0119] Diffusion models have achieved great success in video and image generation. However, due to the large number of parameters and inference steps involved, diffusion models still suffer from significant latency. For example, a common video generation model can take several minutes to generate a 5-second video at 720P resolution. The data processing method proposed in this embodiment can improve the computational efficiency of diffusion models.

[0120] The embodiment of the present application adopts a heterogeneous sparsity allocation method to assign different sparsities to asynchronous and different blocks, so that the entire diffusion model has high accuracy and sparsity. To ensure a high acceleration ratio, the embodiment of the present application adopts block-based structured sparsity and uses channel sorting technology to ensure sparsity and accuracy. To reduce the overhead introduced by sparse prediction, the embodiment of the present application reuses the mask matrix of the previous step, eliminates the prediction process, and skips a certain proportion of the linear calculations of the first and second layers of the Feed-Forward Network (FFN).

[0121] One reason why diffusion models have difficulty increasing sparsity compared to large language models is that they are composed of different Transformer blocks and, during the denoising process, are composed of iterative loops (time steps). First, within the same time step, different Transformer blocks have different sensitivities to sparsity. For example, given the same preset sparsity, most blocks can maintain high accuracy (accuracy refers to whether the output results of the diffusion model using sparsification and not using sparsification differ significantly (a large difference indicates low accuracy)), while a small number of blocks have very poor accuracy. Blocks with very poor accuracy are also sensitive blocks. Second, the sparsity sensitivity of the same Transformer block varies at different time steps. This is because the denoising tasks performed by the asynchronous denoising process are different.

[0122] In order to ensure the quality of the diffusion model generation, the embodiment of the present application proposes a strategy of using heterogeneous sparse allocation, that is, assigning different preset sparsities to different time steps and different Transformer blocks in the diffusion model (i.e., the data processing model), so that all blocks can reach the accuracy threshold during sparse calculation. First, the accuracy distribution of all steps and Transformer blocks is calculated at different sparsities, that is, the cosine similarity (cosine-similarity threshold) between sparse calculation and full-precision calculation; second, given an accuracy threshold (the accuracy threshold can be adjusted according to the final output effect of the diffusion model), the sparsity of all Transformer blocks at all steps is obtained. See Figure 6 , Figure 6 Schematic diagram of the sparsity distribution of different time steps and blocks provided by the embodiment of the present application. It can be seen that the sparsity distribution of different time steps and different blocks calculated using the embodiment of the present application.

[0123] The following is a specific example: For example, a diffusion model M, a set of candidate sparsity S (such as 0.01, 0.1, ..., 0.99), and an accuracy threshold are preset. (i.e. preset similarity threshold). Find the value that keeps the diffusion model performance not lower than the threshold. The maximum sparsity that can be achieved under the premise of . First, initialize the accuracy tensor: , a three-dimensional tensor The dimension is the number of candidate sparsity , time steps and the number of blocks (i.e., data processing modules) . 3D tensor Used to store the accuracy of different sparsities, different time steps and different blocks. Then, traverse each candidate sparsity, and for each candidate sparsity, perform the following steps: traverse each time step and block, for each time step t and each block b, calculate the cosine similarity between the output of the sparse model (i.e., multiple third model output matrices) and the output of the dense model (i.e., the second model output matrix). Cosine similarity is used to measure the directional similarity of two vectors, and here it is used to evaluate the similarity between the outputs of the sparse model and the dense model. In this way, by traversing different time steps and different blocks, the loop of time steps and blocks is finally ended, and then the loop of candidate sparsity is ended. Finally, the optimal sparsity mapping is calculated, and for each time step t and each block b, the accuracy can be found. Greater than or equal to the accuracy threshold The maximum sparsity (i.e., the sparsity threshold) can be stored in the optimal sparsity map, and then the calculated optimal sparsity map is returned.

[0124] In order to improve the acceleration ratio, the embodiment of the present application can perform the following processing for the FFN layer using the SiLU activation function.

[0125] First, a block mask is generated. For the feature tensor after SiLU activation, the size of the feature tensor is M×N. According to the block size m×n (that is, the preset block size), the feature tensor is divided into M / m×N / n sub-blocks. The importance score of each sub-block is Calculated by the following formula (2): (2) in, is the importance score, Indicates the average value, For sub-blocks The absolute value of the elements in , is the feature tensor.

[0126] Then, we apply it to all sub-blocks of the feature tensor X by setting The sparsity threshold (time step t, transformer block b), generates a binary mask matrix The rule is as follows: (3) in, It is a sub-block The mask value of It is a sub-block The importance score of is the sparsity threshold.

[0127] Then, in the sparse calculation stage, the binary mask matrix The blocks marked as 1 (i.e., the first mask value) participate in the matrix multiplication calculation, and the blocks marked as 0 (i.e., the second mask value) are not calculated and directly return the all-0 submatrix.

[0128] Block sparsity methods can make full use of the Tensor Cores in the Graphics Processing Unit (GPU) for efficient algorithm execution. The m×n sub-blocks can be perfectly aligned with the tiled computations of the Tensor Cores. Furthermore, block-granular memory access operations ensure 128-byte aligned memory transfers, significantly improving High Bandwidth Memory (HBM) bandwidth utilization. Compared to unstructured sparsity, the more regular access patterns also significantly improve the L2 cache hit rate, thereby maintaining efficient inference performance. Figure 7 , Figure 7 Schematic diagram of activation value mask distribution under different sparse methods provided in the embodiment of the present application. It can be seen that different sparse methods produce different mask distributions. Figure 7 Part (a) uses the common sparse method (block size is 1) for sparseness. Figure 7 Part (b) uses direct block sparse method for sparseness. Figure 7 Part (c) uses block sparseness plus channel sorting method for sparseness. Figure 7 Part (a) may result in some unimportant features being retained. Figure 7 Part (b) in

[15] can better utilize the correlation between features, but may cause some important features to be mistakenly deleted. Figure 7 Part (c) in

[15] can more effectively utilize the correlation between features and better retain important features.

[0129] The aforementioned block sparsification method has significant accuracy issues in the diffusion model. This is because the calculation of the importance score involves finding the absolute average. That is, when important activation values (larger values in the feature tensor X) and a large number of small values (smaller values in the feature tensor X) are in the same sub-block, the importance score of the sub-block is less than the sparsity threshold, so the sub-block is skipped and important features are easily mistakenly deleted, resulting in a decrease in the accuracy of the diffusion model. To solve this problem, the channel sorting technology provided in the embodiment of the present application can re-aggregate the channels and group smaller values together, thereby preventing the sub-blocks containing important activation values from being judged as blocks that can be sparsely calculated during the block sparsification process, thereby significantly reducing accuracy.

[0130] Specifically, the embodiment of the present application provides a static channel sorting method, which uses data calibration to obtain the importance of different channels. First, a batch of calibration data (i.e., first calibration data) is obtained, which can be obtained by randomly sampling from real data. For each channel c of the feature tensor X (i.e., the second activated feature vector) after SiLU activation, the absolute average of all corresponding values in the single channel is used as the channel importance score. , as shown in the above formula (1), which will not be repeated here. Reorder channels from small to large.

[0131] Figure 7 Part (b) shows the distribution of activation values after channel sorting. The distribution of smaller activation values is aggregated in a closer area, which is conducive to subsequent sparse operations.

[0132] See also Figure 8 , Figure 8 This is a schematic diagram of the channel sorting provided in an embodiment of the present application. The size of the feature tensor is Token×Chan_IN, where Token represents the number of samples and Chan_IN represents the number of input channels. The first linear layer 801 converts the feature tensor into the output feature tensor Y1 through the weight matrix W1. The second linear layer 802 converts the output feature tensor Y1 into the final output feature tensor Y2 through the weight matrix W2. The embodiment of the present application can rearrange the weight matrix W1 through channel sorting. Calculation is performed using the rearranged weight matrix W1 so that the output feature tensor Y1 can also be rearranged. For the output feature tensor Y1, each column is a channel, and 1, 2, 7, and 9 are channel importance scores. It can be seen that each channel has a channel importance score. The rearrangement vector P (i.e., the activation value channel rearrangement vector) is the mapping vector that maps 7, 9, 2, and 1 to 1, 2, 7, and 9.

[0133] However, directly introducing the channel reordering operation of activation values in the FFN layer of the Transformer block will bring a large computational burden. The direct channel sorting process is as shown in formula (4): (4) in, is the rearrangement function, Indicates the use of rearranged vectors , to reorder the matrix , It is obtained by sorting the feature tensor X (i.e., the second activated feature vector) from small to large channels.

[0134] However, performing channel sorting on the activation values of the first linear layer in FFN will introduce two high-cost activation value channel reordering operations, which seriously affects the inference efficiency. In order to solve this problem, the embodiment of the present application proposes a method to eliminate the channel sorting overhead during inference by introducing the channel sorting overhead into the offline weight transformation stage. Figure 8 The figure shows the channel sorting process based on weight transformation. Specifically, before model inference, the diffusion model is transformed as shown in formulas (5)-(6): (5) (6) After the transformation, the forward calculation can be simplified to formula (7): (7) in, is the diffusion model output, is the feature tensor, is the rearrangement weight, is the rearrangement weight, is the mapping vector, is the original weight, is the original weight, is the rearrangement function.

[0135] It can be seen that according to Reorder and according to After reordering, directly perform matrix multiplication, and the original calculation result consistent.

[0136] Sparse methods often require mask calculation and index construction, which will bring high preprocessing overhead. To reduce this overhead, the present embodiment of the application is based on an in-depth analysis of the diffusion model and found that the diffusion model shows a highly consistent sparsity distribution pattern between adjacent time steps. For example, the similarity of the sparse masks in the fifth step and the fourth step is as high as 95%. Figure 9 , Figure 9 This is a schematic diagram of the principle of stride mask multiplexing provided by the embodiment of the present application. Figure 9 The stride mask reuse strategy shown in the figure achieves low computational complexity and low latency mask acquisition. The specific process is as follows: First, at a specified time step (e.g., step zero), the inference of the first linear layer 901 (Layer 1) and the activation of SiLu (activation layer 902) are performed once to calculate the importance score of each block. , referring to formula (8), the corresponding sparse mask can be generated , when calculating the second linear layer 903, a sparse mask can be used Then, at each subsequent time step In , the reference importance score of the initial step can be directly reused. Then, according to the current Transformer block b at the current time step t, the sparsity threshold obtained offline in the previous step is , generate the corresponding sparse mask , as shown in formula (8): (8) in, is the mask value of each block, is the importance score of each block, is the sparsity threshold Then, using the generated mask Perform sparse calculation on the two linear layers of FFN: For the first linear layer of FFN, the mask reveals the sparsity of the output block, indicating that the corresponding input multiplication calculation can be skipped. Output block , directly skip the calculation of formula (9): (9) in, is the output block, is a sub-block, is the weight parameter corresponding to the first linear layer. It can be seen that each time a zero block is skipped, floating-point calculations.

[0137] For the second linear layer of FFN, the mask marks the sparse input block, indicating that the multiplication and addition operation with the corresponding weight can be skipped. Specifically, for Input block , skip the calculation of formula (10): (10) in, is the input channel dimension of the weight, is the output result, is the input block, is the weight parameter corresponding to the second linear layer. Each time a zero block is skipped, floating-point calculations.

[0138] In addition, the embodiment of the present application uses block granularity to store sparse masks. Compared with element-level masks, the memory mask storage overhead can be reduced by 2-3 orders of magnitude, greatly improving memory efficiency. At the same time, the present invention designs an adaptive mask cache refresh mechanism, which recalculates the mask every K steps to ensure that the reused mask keeps synchronization with the feature changes, thereby improving the accuracy brought by reuse.

[0139] This embodiment of the application includes two stages: offline weight transformation and online sparse inference. The specific process is as follows: In the offline phase, first, offline data calibration is performed. The feature tensor X obtained during inference can be used with the calibration data (i.e., the second calibration data) to determine the sparsity threshold under different time steps and different transformer block conditions through statistical means. and channel importance scores Then, the offline weight channel is rearranged and the channel importance is obtained. , through formula (5)-(6), the weight and Rearrange to get and .

[0140] In the online inference phase, for each time step t, it is possible to determine whether the sparse mask needs to be refreshed at the current time step. When t = 0 or t is an integer multiple of the mask refresh interval K (i.e., the current step is an iterative update step), the complete FFN first linear layer activation calculation is performed. , and split the feature tensor into blocks of size m×n, and calculate the importance score of each block , and update the cache backup of the importance score. If the current step time does not need to be refreshed (that is, the current step is a non-iterative update step), the cache backup of the previous importance score is directly reused. According to the sparsity rate and sparsity threshold of the current transformer block at the current time step. Then, the corresponding sparse mask is generated by the above formula (8) In the mask After confirmation, enter the sparse calculation stage. First, perform the sparse FFN first linear layer calculation: for each output block ,like , then execute ; If the mask value is 0, skip the block calculation and The output of is set to zero. Then, the sparse FFN second linear layer calculation is performed: traverse each input block ,like , then matrix multiplication is performed ; Otherwise, skip the accumulation operation of this block. Finally, the output As the sparse inference result of this step size.

[0141] Through the above process, the embodiment of the present application significantly reduces the floating-point calculation amount and memory access overhead while ensuring the accuracy of sparse reasoning, greatly improving the computational efficiency of the diffusion model in the reasoning stage.

[0142] Based on the data processing method described in the above embodiment, Figure 10 A structural block diagram of a data processing device provided in an embodiment of the present application is shown. The data processing device 100 can be a device in an electronic device (for example, a server). The data processing device can be implemented in software, which can be software in the form of programs and plug-ins, etc., including the following software modules: a data acquisition module 101, a data processing module 102 and a data output module 103. These modules are logical and can be arbitrarily combined or further split according to the functions implemented.

[0143] Among them, the data acquisition module 101 is used to obtain the sparsity threshold set of the data to be processed and the data processing model; wherein, the data processing model includes multiple data processing modules, each data processing module includes a first linear layer, an activation layer and a second linear layer; the sparsity threshold set includes: the sparsity threshold of each data processing module in the data processing model at the time step corresponding to each round of iterative processing; the data processing module 102 is used to input the data to be processed into the data processing model, and perform multiple rounds of iterative processing on the data to be processed through the data processing model to obtain a first model output matrix; wherein, in at least one round of iterative processing, a mask matrix is determined based on the first activation feature vector output by the activation layer and the sparsity threshold, and the mask matrix is used to perform sparse calculation in the current step iterative processing and the multiple steps of iterative processing after the current step; the data output module 103 is used to determine the data processing result of the data to be processed based on the first model output matrix.

[0144] In some embodiments, the data acquisition module 101 is also used to obtain a channel importance score set of the data processing model; the channel importance score set includes the channel importance score of each channel in the activation layer; based on the channel importance score of each channel, the activation layer in the data processing model is channel-rearranged to obtain the model parameters after channel rearrangement; the data to be processed is input into the data processing model, and the data processing model is used to perform multiple rounds of iterative processing on the data to be processed to obtain a first model output matrix, including: inputting the data to be processed into the channel rearranged data processing model corresponding to the channel-rearranged model parameters, and the channel rearranged data processing model is used to perform multiple rounds of iterative processing on the data to be processed to obtain the first model output matrix.

[0145] In some embodiments, the data acquisition module 101 is also used to obtain first calibration data; input the first calibration data into the data processing model, and perform linear processing and activation processing on the first calibration data in sequence through the first linear layer and activation layer of the data processing model to obtain a second activation feature vector; determine the absolute average value of the elements corresponding to each channel of the activation layer in the second activation feature vector; determine the absolute average value of each channel as the channel importance score of the channel, and obtain the channel importance score set.

[0146] In some embodiments, the data acquisition module 101 is also used to obtain the weight parameters of each channel in each activation layer in the data processing model; sort the channel feature vectors of the second activation feature vector in order from small to large according to the channel importance score to obtain the activation value channel rearrangement vector; use the activation value channel rearrangement vector to perform weight transformation on the weight parameters of each channel in the activation layer to obtain the model parameters after channel rearrangement.

[0147] In some embodiments, the data acquisition module 101 is also used to input the second calibration data into the data processing model, perform N rounds of iterative processing on the second calibration data through the data processing model to obtain a second model output matrix; obtain a candidate sparsity set; for the Kth round of iterative processing, perform sparse processing on each data processing module in the data processing model according to each candidate sparsity in the candidate sparsity set to obtain multiple sparse models corresponding to each candidate sparsity; K is an integer greater than 0 and less than or equal to N; input the second calibration data into each sparse model to obtain multiple third model output matrices; based on the multiple third model output matrices and the second model output matrix, determine the sparsity threshold set of the data processing model from the candidate sparsity set.

[0148] In some embodiments, the data acquisition module 101 is also used to determine the cosine similarity between each third model output matrix and the second model output matrix; based on the cosine similarity, determine the target sparsity of each data processing module during the K-th round of iterative processing from the candidate sparsity set; use the target sparsity as the sparsity threshold of the data processing module during the K-th round of iterative processing, and store it in the sparsity threshold set.

[0149] In some embodiments, the data acquisition module 101 is also used to, for each data processing module during the K-th round of iterative processing, in response to determining at least one target cosine similarity greater than a preset similarity threshold from all cosine similarities corresponding to the candidate sparsities in the candidate sparsity set, determine the maximum candidate sparsity as the target sparsity of the data processing module during the K-th round of iterative processing from at least one candidate sparsity corresponding to the at least one target cosine similarity; in response to not determining a target cosine similarity greater than the preset similarity threshold from all cosine similarities corresponding to the candidate sparsities in the candidate sparsity set, set the target sparsity of the data processing module during the K-th round of iterative processing to a preset value.

[0150] In some embodiments, the data processing module 102 is further used to, in response to determining that the current step is an iterative update step based on the number of iterations of the current step, obtain the first activation feature vector output by the activation layer under the current step and the sparsity threshold of the data processing module where the activation layer is located under the current step; block the first activation feature vector according to a preset block size to obtain a plurality of sub-blocks; determine the importance score of each sub-block based on the elements in each sub-block; in response to the importance score of any sub-block being greater than or equal to the sparsity threshold, determine the mask value corresponding to the sub-block in the mask matrix as the first mask value; in response to the importance score of any sub-block being less than the sparsity threshold, determine the mask value corresponding to the sub-block in the mask matrix as the second mask value.

[0151] In some embodiments, the data processing module 102 is further used to, in response to determining that the current step is a non-iterative update step based on the iterative step number of the current step, obtain the mask matrix determined in the most recent iterative update step before the current step; and determine the mask matrix determined in the most recent iterative update step before the current step as the mask matrix of the current step.

[0152] In some embodiments, the data processing module 102 is further configured to, during each round of iterative processing, for each data processing module in the data processing model, perform a first linear processing on the input feature through the first linear layer of the data processing module based on the mask matrix of the data processing module to obtain a first linear feature vector; wherein, for the first data processing module in the first round of iterative processing, the input feature is the data to be processed; for the first data processing module in the i-th round of iterative processing, the input feature is the model output matrix in the i-1-th round of iterative processing; for the j-th data processing module in the i-th round of iterative processing, the input feature is the model output matrix in the i-th round of iterative processing The output eigenvector of the j-1th data processing module at this time; wherein, i is an integer greater than 1 and less than or equal to N; j is an integer greater than 1 and less than or equal to L, and L is the total number of data processing modules in the data processing model; the first linear eigenvector is activated by the activation layer of the data processing module to obtain a third activated eigenvector; based on the mask matrix of the data processing module, the third activated eigenvector is subjected to a second linear processing by the second linear layer of the data processing module to obtain the output eigenvector of the data processing module; the model output matrix of the last round of iterative processing is determined as the first model output matrix obtained after the multiple rounds of iterative processing.

[0153] In some embodiments, the data processing module 102 is further used to perform block segmentation on the input features according to a preset block size to obtain multiple input blocks; determine the mask value of each input block from the mask matrix of the data processing module; in response to the mask value of any input block being the second mask value, skip the weight calculation of the input block; in response to the mask value of any input block being the first mask value, perform weighted summation on the input blocks based on the weight parameters of the input blocks to obtain the first linear feature vector.

[0154] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0155] An embodiment of the present application provides an electronic device, Figure 11 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 11 As shown, the electronic device 130 includes: at least one processor 131 ( Figure 11 Only one is shown in the figure), a memory 132, and computer executable instructions 133 stored in the memory 132 and executable on at least one processor 131. When the processor 131 executes the computer executable instructions 133, the steps of any of the above-mentioned data processing method embodiments are implemented.

[0156] The electronic device may include but is not limited to a processor 131 and a memory 132. It will be understood by those skilled in the art that Figure 11 This is merely an example of the electronic device 130 and does not constitute a limitation on the electronic device 130 . The electronic device 130 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0157] The processor 131 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPG), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0158] In some embodiments, the memory 132 may be an internal storage unit of the electronic device 130, such as a hard drive or memory of the electronic device 130. In other embodiments, the memory 132 may also be an external storage device of the electronic device 130, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card, etc. equipped on the electronic device 130. Furthermore, the memory 132 may include both an internal storage unit of the electronic device 130 and an external storage device. The memory 132 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of a computer program. The memory 132 may also be used to temporarily store data that has been output or is about to be output.

[0159] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data processing method described in the embodiment of the present application.

[0160] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the data processing method provided in the embodiment of the present application, for example, Figure 1 The data processing method is shown.

[0161] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0162] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0163] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0164] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0165] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a set of sparsity thresholds for data to be processed and a data processing model; wherein the data processing model includes a plurality of data processing modules, each of which includes a first linear layer, an activation layer, and a second linear layer; and the sparsity threshold set includes: a sparsity threshold for each data processing module in the data processing model at a time step corresponding to each round of iterative processing; Inputting the data to be processed into the data processing model, performing multiple rounds of iterative processing on the data to be processed by the data processing model to obtain a first model output matrix; wherein, during at least one round of iterative processing, determining a mask matrix based on the first activated eigenvector output by the activation layer and the sparsity threshold, the mask matrix being used to perform sparse calculations during the current iterative processing step and multiple iterative processing steps after the current step; A data processing result of the data to be processed is determined based on the first model output matrix.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining a channel importance score set of the data processing model; the channel importance score set includes a channel importance score of each channel in the activation layer; Based on the channel importance score of each channel, performing channel rearrangement on the activation layer in the data processing model to obtain model parameters after channel rearrangement; The step of inputting the data to be processed into the data processing model and performing multiple rounds of iterative processing on the data to be processed by the data processing model to obtain a first model output matrix includes: The data to be processed is input into a channel rearrangement data processing model corresponding to the model parameters after the channel rearrangement, and the data to be processed is iteratively processed for multiple rounds by the channel rearrangement data processing model to obtain a first model output matrix.

3. The method according to claim 2, characterized in that The obtaining of the channel importance score set of the data processing model includes: Acquiring first calibration data; Inputting the first calibration data into the data processing model, and sequentially performing linear processing and activation processing on the first calibration data through a first linear layer and an activation layer of the data processing model to obtain a second activation feature vector; Determining the absolute average of the elements corresponding to each channel of the activation layer in the second activation feature vector; The absolute average value of each channel is determined as the channel importance score of the channel to obtain the channel importance score set.

4. The method according to claim 3, characterized in that The channel importance score of each channel is used to perform channel rearrangement on the activation layer in the data processing model to obtain model parameters after channel rearrangement, including: For each activation layer in the data processing model, obtaining a weight parameter of each channel in the activation layer; Sorting the second activation feature vectors by channel feature vectors in ascending order of the channel importance scores to obtain an activation value channel rearrangement vector; The activation value channel rearrangement vector is used to perform weight transformation on the weight parameters of each channel in the activation layer to obtain model parameters after channel rearrangement.

5. The method according to claim 1, characterized in that Get the sparsity threshold set for the data processing model, including: Inputting the second calibration data into the data processing model, and performing N rounds of iterative processing on the second calibration data through the data processing model to obtain a second model output matrix; Get the candidate sparsity set; For the Kth round of iterative processing, according to each candidate sparsity in the candidate sparsity set, each data processing module in the data processing model is subjected to sparse processing to obtain multiple sparse models corresponding to each candidate sparsity; K is an integer greater than 0 and less than or equal to N; Inputting the second calibration data into each sparse model to obtain a plurality of third model output matrices; Based on the multiple third model output matrices and the second model output matrix, a sparsity threshold set of the data processing model is determined from the candidate sparsity set.

6. The method according to claim 5, characterized in that The step of determining a sparsity threshold set of the data processing model from the candidate sparsity set based on the plurality of third model output matrices and the second model output matrix comprises: determining a cosine similarity between each third model output matrix and the second model output matrix; Based on the cosine similarity, determining the target sparsity of each data processing module in the K-th round of iterative processing from the candidate sparsity set; The target sparsity is used as the sparsity threshold of the data processing module during the K-th round of iterative processing and is stored in the sparsity threshold set.

7. The method according to claim 6, characterized in that The step of determining the target sparsity of each data processing module in the K-th round of iterative processing from the candidate sparsity set based on the cosine similarity includes: For each data processing module during the K-th round of iterative processing, In response to determining at least one target cosine similarity greater than a preset similarity threshold from all cosine similarities corresponding to candidate sparsities in the candidate sparsity set, determining a maximum candidate sparsity from at least one candidate sparsity corresponding to the at least one target cosine similarity as the target sparsity of the data processing module during the Kth round of iterative processing; In response to not determining a target cosine similarity greater than a preset similarity threshold from all cosine similarities corresponding to candidate sparsities in the candidate sparsity set, setting the target sparsity of the data processing module in the Kth round of iterative processing to a preset value.

8. The method according to claim 1, characterized in that The step of determining a mask matrix based on the first activated feature vector output by the activation layer and the sparsity threshold during at least one round of iterative processing includes: In response to determining that the current step is an iterative update step based on the number of iterations of the current step, obtaining a first activation feature vector output by the activation layer at the current step and a sparsity threshold of a data processing module where the activation layer is located at the current step; Performing block segmentation on the first activated feature vector according to a preset block size to obtain a plurality of sub-blocks; Determine an importance score for each sub-block based on the elements in each sub-block; In response to an importance score of any sub-block being greater than or equal to the sparsity threshold, determining a mask value corresponding to the sub-block in the mask matrix as a first mask value; In response to the importance score of any sub-block being smaller than the sparsity threshold, a mask value corresponding to the sub-block in the mask matrix is determined as a second mask value.

9. The method according to claim 8, characterized in that The method further comprises: In response to determining that the current step is a non-iterative update step based on the number of iterations of the current step, obtaining a mask matrix determined in a most recent iterative update step before the current step; The mask matrix determined in the most recent iterative update step before the current step is determined as the mask matrix of the current step.

10. The method according to any one of claims 1 to 9, characterized in that The step of performing multiple rounds of iterative processing on the data to be processed by the data processing model to obtain a first model output matrix includes: In each round of iterative processing, for each data processing module in the data processing model, Based on the mask matrix of the data processing module, a first linear processing is performed on the input feature through the first linear layer of the data processing module to obtain a first linear feature vector; wherein, for the first data processing module in the first round of iterative processing, the input feature is the data to be processed; for the first data processing module in the i-th round of iterative processing, the input feature is the model output matrix in the i-1-th round of iterative processing; for the j-th data processing module in the i-th round of iterative processing, the input feature is the output feature vector of the j-1-th data processing module in the i-th round of iterative processing; wherein i is an integer greater than 1 and less than or equal to N; j is an integer greater than 1 and less than or equal to L, and L is the total number of data processing modules in the data processing model; Performing activation processing on the first linear feature vector through the activation layer of the data processing module to obtain a third activated feature vector; Based on the mask matrix of the data processing module, performing a second linear processing on the third activated eigenvector through a second linear layer of the data processing module to obtain an output eigenvector of the data processing module; The model output matrix of the last round of iterative processing is determined as the first model output matrix obtained after the multiple rounds of iterative processing.

11. The method according to claim 10, characterized in that The step of performing a first linear processing on the input feature based on the mask matrix of the data processing module through the first linear layer of the data processing module to obtain a first linear feature vector includes: Performing block segmentation on the input feature according to a preset block size to obtain a plurality of input blocks; determining a mask value for each input block from a mask matrix of the data processing module; In response to the mask value of any input block being the second mask value, skipping weight calculation for the input block; In response to a mask value of any input block being a first mask value, weighted summation is performed on the input blocks based on a weight parameter of the input block to obtain the first linear feature vector.

12. A data processing device, characterized in that: The device comprises: A data acquisition module, configured to acquire data to be processed and a set of sparsity thresholds of a data processing model; wherein the data processing model includes a plurality of data processing modules, each of which includes a first linear layer, an activation layer, and a second linear layer; and the set of sparsity thresholds includes: a sparsity threshold of each data processing module in the data processing model at a time step corresponding to each round of iterative processing; a data processing module, configured to input the data to be processed into the data processing model, and perform multiple rounds of iterative processing on the data to be processed through the data processing model to obtain a first model output matrix; wherein, during at least one round of iterative processing, a mask matrix is determined based on the first activated eigenvector output by the activation layer and the sparsity threshold, and the mask matrix is used to perform sparse calculations during the current iterative processing step and multiple iterative processing steps after the current step; A data output module is used to determine the data processing result of the data to be processed based on the first model output matrix.

13. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; The processor is configured to implement the data processing method according to any one of claims 1 to 11 when executing the computer-executable instructions or computer programs stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.

Images