A large model online pruning method for cross-domain data heterogeneity and related equipment

Abstract

The present invention discloses a method for online pruning of a large model with cross-domain data heterogeneity and related equipment. The method comprises the following steps: obtaining a target pre-trained model and a target data set; pre-pruning the target pre-trained model according to the target data set to obtain a pre-pruned model; sending the parameters of the pre-pruned model to a cloud server for processing to obtain target fusion parameters, loading the target fusion parameters into the pre-pruned model to obtain a target fusion model, and updating the target pre-trained model to a target fusion model; re-executing the step of pre-pruning the target pre-trained model according to the target data set until a preset number of rounds are completed, and then using the target fusion model as the target pre-pruned model; performing structural pruning on the target pre-pruned model to obtain a target pruned model; and obtaining a target release model based on the target pruned model. The present invention can perform online pruning of models in a cross-domain network environment, reduce model parameters, and lower the hardware requirements for large model deployment.

Description

Technical Field

[0001] The present invention relates to the field of model pre-training technology, and in particular to an online pruning method for a large model with cross-domain heterogeneous data and related equipment. Background Art

[0002] Large Transformer-based pre-trained models for language, image, and multimodal tasks have achieved significant success across a wide range of evaluation tasks, demonstrating the superiority of the Transformer architecture based on multi-head attention modules. However, these models typically have hundreds of millions or even tens of billions of parameters, placing significant demands on hardware for deployment in real-world environments.

[0003] Therefore, the existing technology still needs to be improved and enhanced. Summary of the Invention

[0004] In response to the above-mentioned defects of the prior art, the present invention provides a large-scale online pruning method for cross-domain data heterogeneity and related equipment, aiming to solve the problems of large model parameters and high hardware requirements in the prior art of pre-training large models.

[0005] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0006] A first aspect of the present invention provides a method for online pruning of a large model with heterogeneous cross-domain data, the method comprising:

[0007] Obtain the target pre-trained model and target dataset;

[0008] Pre-pruning the target pre-trained model according to the target data set to obtain a pre-pruned model;

[0009] Sending the parameters of the pre-pruned model to a cloud server for processing to obtain target fusion parameters, loading the target fusion parameters into the pre-pruned model to obtain a target fusion model, and updating the target pre-trained model to the target fusion model;

[0010] Re-performing the step of pre-pruning the target pre-trained model according to the target data set until a preset number of rounds are completed, and then using the target fusion model as the target pre-pruned model;

[0011] Performing structural pruning on the target pre-pruned model to obtain a target pruned model;

[0012] A target publishing model is obtained according to the target pruning model.

[0013] The target pre-training model is a transformer model.

[0014] Before obtaining the target pre-trained model and the target dataset, the method further includes:

[0015] Acquire an original data set, and preprocess the original data to obtain the target data set;

[0016] The target dataset template is consistent with the preset template.

[0017] The pre-pruning of the target pre-trained model according to the target data set includes:

[0018] Obtain the multi-head attention module in the target pre-trained model;

[0019] Setting a corresponding gating network according to each head in the multi-head attention module;

[0020] Perform a dot product between the output of the multi-head attention module corresponding to each training data in the target data set and the gated network to obtain the target attention output;

[0021] A pre-pruning training loss is obtained according to the gated sparsification function and the target attention output, and the target pre-training model is updated according to the optimization target.

[0022] The gated sparsification function is:

[0023]

[0024] Among them, the number of layers of the multi-head attention module in the target pre-training model is l, G i is the gating network corresponding to the multi-layer attention module in the i-th layer, i∈[0,l).

[0025] The pre-pruning training loss is:

[0026]

[0027] Among them, θ is the original transformer model parameter, are the parameters of the gating network part, is the cross entropy loss function of the target pre-trained model, is the gated sparsity loss function, and λ is a hyperparameter.

[0028] The target fusion parameters are obtained by the cloud server calculating the weighted average of the non-gated network part model parameters in the parameters of the pre-pruned models of different computing centers, and performing an extreme value selection fusion strategy on the gated network part in the pre-pruned model parameters of different computing centers.

[0029] The performing structural pruning on the target pre-pruning model includes:

[0030] Performing binarization approximation processing on the gating value in the gating network to obtain a binarized gating value, wherein the binarized gating value is 0 or 1;

[0031] Structural pruning is performed on the target pre-pruned model according to the binarized gating value to obtain a target pruning model.

[0032] The performing binarization approximation processing on the gating value in the gating network to obtain the binarized gating value includes:

[0033] Using the first formula to obtain the binarized gating value;

[0034] The first formula is

[0035]

[0036]

[0037]

[0038] Wherein, z is the binarized gate value, u is the random noise that obeys the uniform distribution of U(0,1), g j For the i-th gating network G i The jth element of , β is a hyperparameter, s is the distribution of the gating network mapped to the (0,1) interval after adding noise, is the intermediate parameter, γ and ζ are parameters, γ<0,ζ>1.

[0039] The performing structural pruning on the target pre-pruning model according to the binarized gating value includes:

[0040] The attention heads in the multi-head attention module corresponding to the binary gating value of 0 are pruned.

[0041] A second aspect of the present invention provides an online pruning device for a large model with heterogeneous cross-domain data, comprising:

[0042] A data acquisition module, wherein the acquisition module is used to acquire a target pre-training model and a target data set;

[0043] A pre-pruned model acquisition module, configured to pre-prune the target pre-trained model according to the target data set to obtain a pre-pruned model;

[0044] A fusion model acquisition module is used to send the pre-pruned model parameters to the cloud server for processing to obtain target fusion parameters, load the target fusion parameters into the pre-pruned model to obtain a target fusion model, and update the target pre-trained model to the target fusion model;

[0045] An updating module, configured to re-execute the step of pre-pruning the target pre-trained model according to the target data set until a preset number of rounds are completed, and then use the target fusion model as the target pre-pruned model;

[0046] A pruning model acquisition module, configured to perform structural pruning on the target pre-pruning model to obtain a target pruning model;

[0047] A release model acquisition module is used to obtain a target release model based on the target pruning model.

[0048] In a third aspect of the present invention, a terminal is provided, comprising a processor and a computer-readable storage medium communicatively connected to the processor, wherein the computer-readable storage medium is suitable for storing a plurality of instructions, and the processor is suitable for calling the instructions in the computer-readable storage medium to execute the steps of the online pruning method for a large model with cross-domain data heterogeneity as described above.

[0049] In a fourth aspect of the present invention, a computer-readable storage medium is provided, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the online pruning method for large models with cross-domain data heterogeneity as described in any of the above items.

[0050] Compared with the prior art, the present invention provides a large model online pruning method with cross-domain data heterogeneity. The large model online pruning method with cross-domain data heterogeneity obtains a target pre-trained model and a target data set; pre-prunes the target pre-trained model according to the target data set to obtain a pre-pruned model; sends the parameters of the pre-pruned model to a cloud server for processing to obtain target fusion parameters, loads the target fusion parameters into the pre-pruned model to obtain a target fusion model, and updates the target pre-trained model to the target fusion model; re-executes the step of pre-pruning the target pre-trained model according to the target data set until a preset number of rounds are completed, and then uses the target fusion model as the target pre-pruned model; performs structural pruning on the target pre-pruned model to obtain a target pruned model; and obtains a target release model based on the target pruned model. The present invention can perform online pruning on large models in a cross-domain network environment, reduce model parameters, and lower the threshold for hardware requirements for large model deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flowchart of an embodiment of the online pruning method for a large model with cross-domain data heterogeneity provided by the present invention;

[0052] Figure 2This is an architectural diagram of an embodiment of the online pruning method for a large model with cross-domain data heterogeneity provided by the present invention;

[0053] Figure 3 A diagram of a gated network cross-domain fusion method for an embodiment of the cross-domain data heterogeneity large model online pruning method provided by the present invention;

[0054] Figure 4 A schematic diagram showing the structure of an embodiment of the large-scale online pruning device for cross-domain data heterogeneity provided by the present invention;

[0055] Figure 5 This is a schematic diagram of the principles of an embodiment of a terminal provided by the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] Those skilled in the art will appreciate that, unless expressly stated otherwise, the singular forms '"a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0058] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0059] Example 1

[0060] The online pruning method for a large model with heterogeneous cross-domain data provided in this embodiment can be applied to a terminal with computing capabilities. The terminal can execute the online pruning method for a large model with heterogeneous cross-domain data provided by the present invention to perform online pruning tasks on a large model of cross-domain data.

[0061] Considering the scenario of a multi-domain domestic network of computing centers, this embodiment proposes a new model parameter compression paradigm for online pruning of large models across multiple domains. Private data from each computing center does not need to be transferred locally. Instead, the locally trained model parameters are distributed. Through collaborative training parameter fusion, the knowledge gained from training large models across heterogeneous data domains is integrated, while the large model structure is dynamically and adaptively pruned. The pruned model is then distributed to each computing center domain. Each domain then performs local fine-tuning using a small batch of data to obtain the final data-sensitive compressed model.

[0062] To fully utilize domain-specific datasets in real-world, multi-domain network environments, and jointly compress large models, this embodiment proposes an online pruning method for large models with heterogeneous cross-domain data. This method effectively and dynamically prunes and compresses large models of 2 billion or more data points in a multi-domain network environment. Ultimately, a compressed model is generated in each domain, and the compressed model is more accurate than the result obtained by local compression in each domain.

[0063] The method described in this embodiment can form a rich model ecosystem in a cross-domain network environment, and the number of compressed model parameters is significantly reduced, which can lower the hardware requirements for large model deployment and reduce model inference latency.

[0064] This embodiment is an online pruning method for large models with cross-domain data heterogeneity. When tracking a target, a single target tracking model based on a combination of Transformer and cross-correlation is constructed. The model takes the search region and template as input. The two branches first extract features through a backbone network with the same structure, and then further extract and fuse features through a Transformer combined with cross-correlation. Finally, the two branches output classification and regression predictions respectively. The final result is screened by the size of the classification value, and the final tracking result is generated after post-processing.

[0065] like Figure 1 As shown, in one embodiment of the cross-domain data heterogeneity large model online pruning method provided by the present invention, the cross-domain data heterogeneity large model online pruning includes the steps of:

[0066] S100: Obtain a target pre-trained model and a target dataset.

[0067] The target pre-training model is a large transformer model.

[0068] At present, a large number of artificial intelligence computing centers have been established in China, which have large-scale cluster computing resources. Under the condition that each computing center reaches an agreement, a simple way to achieve computing power networking and complete collaborative computing tasks is, for example, to build a public network server, and each computing center sends valid information to the public network server (in this embodiment, it is defined as a cloud server). After receiving messages from a certain number of computing centers, the cloud server integrates and processes the messages and then distributes them to each computing center, thus completing a round of cross-domain collaborative training.

[0069] As Figure 2 shown, the cloud server is connected to each computing center through a network. The cloud server distributes the same target pre-trained model to each computing center. Each computing center participant receives the target pre-trained model, and on the premise of the combined computing power network, each computing center participant negotiates and first preprocesses the local original data set according to the rule consistent with the same category task template and selects a suitable template.

[0070] Specifically, each computing center exchanges information through the cloud server. Before starting cross-domain large model online pruning, it is necessary to preprocess the local original data of each computing center.

[0071] Among them, preprocessing the local original data set includes:

[0072] Obtain the original data set, preprocess the original data to obtain the target data set;

[0073] The target data set template is consistent with the preset template.

[0074] Specifically, in the data preprocessing stage, before the cross-domain training task is initiated, it is usually necessary for each participant to negotiate, so as to generally understand the distribution of the private data of each computing center. After negotiation, each computing center first judges and classifies its local data set into the specified task categories and agrees on the data templates for each category of tasks.

[0075] For example, when the private data set of computing center 1 is the Chinese reading comprehension data set CMRC2017 and the private data set of computing center 2 is the People's Daily PD cloze data set, the goal is to combine the computing power and data of the two parties to achieve cross-domain knowledge integration and improve the performance of the pruned model. The sample preprocessing results of computing center 1 are as follows:

[0076] Suppose the original data in computing center 1 is:

[0077] <CMRC2017 original data set segment>:

[0078] 1. During the Spring and Autumn Period, Gou Jian, the king of Yue, was defeated by Fu Chai, the king of Wu. He was imprisoned in a certain place for three years and suffered great humiliation. After returning to his country, he was determined to strengthen himself and strive to restore his kingdom.

[0079] 2. Ten years passed. The state of Yue became prosperous and strong, with a large population and powerful military forces. The soldiers once again came to Gou Jian to request a battle: "My lord, the people of the four directions in Yue love you as much as they love their own parents."

[0080] <qid_0>During the Spring and Autumn Period, Gou Jian, the king of Yue, was defeated by Fu Chai, the king of Wu. He was imprisoned in X for three years and suffered great humiliation. After returning to his country, he was determined to strengthen himself and strive to restore his kingdom. (Here, X is the part to be filled in).

[0081] <Corresponding data label>:

[0082] Wu

[0083] The target data after preprocessing the original data through the said data template is: <00​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​

[0098] The target data after preprocessing the original data through the data template is as follows:

[0099] <Data format after PD preprocessing>:

[0100] Fill in the blanks:

[0101] The smart way to invest in the US stock market this year is to follow the "dumb money".

[0102] The so-called "dumb money" X is actually an ordinary portfolio of buying and holding US stocks.

[0103] This strategy works much better than the more complex investment methods used by hedge funds and other professional investors.

[0104] The so-called "dumb money"

[0105] <Data label>:

[0106] Strategy

[0107] Considering the characteristics of the private data sets of the computing center 1 and the computing center 2, they can actually be classified into the general category of cloze tasks. Through the paradigm of continuing to generate after the above text + question content by the model, a unified construction method for two heterogeneous data is realized. And a unified data preprocessing template is designed, and a task template such as "Fill in the blanks: \n" will be added at the beginning of the data; for the part "XXXXX" where filling in the blanks is required in the text content, it is replaced with a filling identifier template such as "()". After adding the "\n" line break character to the above text, the text content of the question is appended. The question content needs to be truncated in front of the question filling place, aiming to enable the model to learn the above knowledge and generate content after the question sentence to complete the filling.

[0108] Such as Party C of the computing center 1 i , plans to use the local data set d i to participate in cross-domain model pruning. First, according to the constraints of the general task category, the local data set d i is classified into a certain general task category t i , and then the corresponding task template of t i is used to preprocess d i into d i '.

[0109] After processing the original data set into the target data set, the following steps are further included:

[0110] S200. Pre-prune the target pre-trained model according to the target data set to obtain a pre-pruned model.

[0111] Large Transformer-based models can scale to billions, tens of billions, or even hundreds of billions of instances. Due to current limitations on graphics card computing resources, these large models are difficult to fit on a single graphics card. Multi-dimensional parallel strategies are often required to split the model across different compute cards. This is then achieved through various MPI-based communication library operators to ensure computational accuracy after the split. Therefore, when performing online pruning on large Transformer models, it is important to consider the slicing of the model structure.

[0112] The multi-head attention module is a crucial component of the large Transformer model. Its computational logic can be fully parallelized. Therefore, when the module's parameters are split across different compute cards, the computational logic remains consistent across each card. Online pruning of large models can be achieved by locally pruning the multi-head attention module shards and incorporating communication operators.

[0113] In this embodiment, pre-pruning the target pre-trained model according to the target data set includes:

[0114] S210, obtaining a multi-head attention module in the target pre-training model;

[0115] S220, setting a corresponding gating network according to each head in the multi-head attention module;

[0116] S230, performing a dot product between the output result of the multi-head attention module corresponding to each training data in the target data set and the gated network to obtain a target attention output;

[0117] S240. Obtain a pre-pruning training loss according to the gated sparsification function and the target attention output, and update the target pre-training model according to the optimization target.

[0118] The optimization goal is to make the gating value in the gating network as small as possible, close to 0.

[0119] In this embodiment, it is assumed that the target pre-trained model has a total of l layers of multi-head attention, the maximum length of the model input is S, the model hidden layer dimension is h, the number of model attention heads is n, and the model parallel number is configured as mp when training a large model. Due to the configuration of the large model parallel dimension mp, for the multi-head attention module: the hidden layer dimension slice of each card is The number of attention heads becomes In the i-th layer of the large model, i∈[0,n), a gate network G is set in the i-th layer multi-head attention. i ,i∈[0,n). G i Parameter shape and number of attention heads in the current attention module slice Consistent, that is

[0120] In the target pre-training model, layer i, i∈[0,n), the gated network G i ,i∈[0,n), and the multi-head attention output result MSA of its corresponding layer output Dot product to obtain the attention feature output after the gated value mask.

[0121] To achieve the effect of pruning the target pre-trained model structure, it is sufficient to change the gate values in the gated network. When the gate value is close to 0, the output of the corresponding attention head after the dot product of the gate value mask is also close to 0, which is equivalent to invalidating the attention head. This invalidation process does not actually reduce the number of model parameters, but achieves a similar effect by setting them to 0.

[0122] In order to make the gated networks G of each layer in the target pre-training model i As sparse as possible, that is, the gated network G i The corresponding gate values g j Close to 0, the g j is the gate value corresponding to the jth attention head in the i-th layer of multi-head attention. In this embodiment, the gated sparsification loss function is:

[0123]

[0124] Among them [[G i =0]] is the indicator function.

[0125] In the gated sparsification loss function (1), [[G i =0]] is the L0 Norm function, which is theoretically not differentiable and needs to be approximated by a continuous function. Therefore, the binarized discrete function of the gated sparsification loss function (1) is approximated as a continuous concrete function (continuous discrete function):

[0126]

[0127] in, is the hyperparameter of the gated sparsification loss function (1) of the approximate continuous discrete function, so that the gate value g in the continuous discrete function (2) is j The distribution is concentrated in the interval [0,1].

[0128] In this continuous discrete function, the sparsity of the gated elements of all layers is summed, defining the prunable structure of the entire target pre-trained model network. By minimizing this continuous discrete function, the gated values of all gated networks are made as close to 0 as possible. The mask is applied to the multi-head attention module, invalidating the corresponding attention head connection structure and approximately pruning the model structure.

[0129] Finally, in the gated sparsity loss function L in the continuous discrete function formula (2) g The loss function set L added to the target pre-trained model raw (θ), that is, the pre-pruning training loss is:

[0130]

[0131] The gated sparsity loss function L of the pre-pruning training loss formula (2) is g Append the original transformer model fine-tuning loss function such as cross entropy L raw (θ), θ is the original transformer model parameter, are the parameters of the gating network part, It is a gated sparsity loss function, and by introducing the λ hyperparameter adjustment, the pre-pruning effect of the target pre-trained model structure is achieved to obtain the pre-pruned model.

[0132] The pre-pruning of the target pre-trained model according to the target data set further includes:

[0133] Perform multiple rounds of training iterations on the target pre-trained model according to the target dataset.

[0134] After obtaining the pre-pruning model, the method further includes the following steps:

[0135] S300: Send the pre-pruned model parameters to the cloud server for processing to obtain target fusion parameters, load the target fusion parameters into the pre-pruned model to obtain a target fusion model, and update the target pre-trained model to the target fusion model.

[0136] Among them, the target fusion parameter is obtained by the cloud server calculating the weighted average of the non-gated network part model parameters in the parameters of the pre-pruned models of different computing centers, and performing an extreme value selection fusion strategy on the gated network part in the pre-pruned model parameters of different computing centers.

[0137] Specifically, the computing center sends the parameters of the pre-pruned model to the cloud server for processing. Taking into account the actual cross-domain scenario, the network bandwidth is limited, usually 10M or 100M interconnection speed, that is, the connection between the computing center and the cloud server is limited. If the computing center and the cloud server exchange large model parameters in each round of training, the frequency of communication is too high and the overhead is huge, which will seriously affect the convergence delay of the online pruning of the large model. Therefore, the experiment uses the AISynergy framework that supports cross-domain training of large models. After each computing center implements multiple rounds of training iterations, the pre-pruned model is obtained. By sending the parameters of the pre-pruned model, the current pre-pruned model is sent to the cloud server via the public network TCP.

[0138] After receiving model parameters from all computing centers, the cloud server performs model parameter fusion. Because each participant's model is trained using differentiated data, the non-gated network model parameters learn from the knowledge of each participant's local data. Different data learning results form gated network parameters with different degrees of sparsity.

[0139] When fusing model parameters, the model parameters are directly averaged to integrate the learning effects of multiple computing centers. In this embodiment, the weighted average of the model parameters of the non-gated network is calculated, and the weighted value is determined by the gating state of both parties: Assume that during the pruning process, the parameter server obtains the gated networks G1, G2, ..., G n , in the gated elements g1,g2,…,g n ,g i ∈G i On the other hand, the minimum value g of the same gating unit of multiple parties is obtained by using the minimum value selection method. min , and then multiply it to the corresponding model parameter slice to achieve a one-to-one mapping between the gating elements and the model parameter slices to fuse the knowledge of the heterogeneous data of each participant; similarly, for the gating network part, an extreme value selection fusion strategy is designed. On each gating element, the minimum value selection method is used to obtain the minimum value of the same gating unit of multiple parties as the gating value after the current step of fusion heterogeneous data training. Figure 3 By using the two-part fusion strategy for the non-gated model parameters and the gated network parameters, the pruning rate of the target pre-pruned model is increased as much as possible, thereby accelerating the convergence speed of the online pruning of the target pre-pruned model.

[0140] After the cloud server completes parameter fusion, it encapsulates the parameters into a message sequence to obtain the target fusion parameters, wherein each of the first fusion models corresponds to one of the computing centers. The cloud server distributes the target fusion parameters to each of the computing centers via TCP. After receiving the target fusion parameters, the computing center calls the relevant interface of the deep learning framework, loads the target fusion parameters into the pre-pruned model, obtains the target fusion model, and updates the target pre-trained model to the target fusion model. Furthermore, according to needs, you can choose to insert the model accuracy verification process and continue model training on local data.

[0141] S400, re-execute the step of pre-pruning the target pre-trained model according to the target data set until a preset number of rounds are completed, and use the target fusion model as the target pre-pruned model.

[0142] After completing the preset number of rounds, the deep learning framework interface is called to save the target pre-pruned model and obtain a pruned model with the data tendency of the target dataset in the computing center.

[0143] Refer again Figure 1 The online pruning method for a large model with heterogeneous cross-domain data in this embodiment further includes the following steps:

[0144] S500: Perform structural pruning on the target pre-pruned model to obtain a target pruned model.

[0145] After obtaining the target pre-pruned model, structural pruning is performed on the target pre-pruned model to obtain a target pruned model.

[0146] The structural pruning process of the target pre-pruning model includes:

[0147] S510, performing binarization approximation processing on the gating value in the gating network to obtain a binarized gating value, wherein the binarized gating value is 0 or 1;

[0148] S520: Perform structural pruning on the target pre-pruned model according to the binarized gating value to obtain a target pruning model.

[0149] The step of performing structural pruning on the target pre-pruning model according to the binary gating value includes:

[0150] The attention heads in the multi-head attention module corresponding to the binary gating value of 0 are pruned.

[0151] In the above process, the gating network sparse loss function will make the model gating value as close to 0 as possible. In order to actually reduce the number of model parameters and reduce the actual memory occupied during model deployment, the gating value needs to be binarized and approximated to (0, 1) to achieve true pruning of the target pre-trained model structure. In this embodiment, the continuous discrete function concrete function is transformed into a hard concrete function, and the gating value approaching 0 is clipped to 0, and the gating value approaching 1 is clipped to 1, thereby achieving binarization approximation of the gating network. Gated network G i Calculate the hard concrete function. The core uses hardsigmoid to obtain the final binary gate value z. The calculation logic formula is the first formula, as follows:

[0152]

[0153]

[0154]

[0155] Among them, u is random noise that obeys the uniform distribution of U(0,1), g j For the i-th gating network G i The algorithm first maps the noised gated network to the s distribution in the (0,1) interval through the sigmoid function of formula (4). is an intermediate parameter. The purpose of adding noise is to make the learning result more robust. Then stretch the s distribution from the (0,1) interval to the (γ,ζ) interval, and we get Distribution, where γ<0,ζ>1, γ is less than but close to 0, ζ is greater than but close to 1, which is an adjustable stretch coefficient. The probability density of the stretching result will change between 0 and 1. By adjusting the results of each parameter, you can make The probability density of the distribution is close to 0, which is more conducive to pruning. Finally, the hard-sigmoid function of formula (6) is used to The distribution clip is changed to 0 or 1 as the final gate value z, which realizes the binarization result of the gated network.

[0156] After calculation, the binary gating network in each multi-head attention module in the pre-pruning model is obtained. The gating values of the gating network are mapped one-to-one to the corresponding attention heads. If the multi-head attention head corresponding to the current gating value needs to be pruned, the multi-head attention module of the current layer is traced back to all operators involved in the calculation of the attention head, and the weights of the operators are mathematically equivalently sliced and inherited.

[0157] For example, in the method of this embodiment, the operator result output is binarized with a 0-1 mask. When corresponding to the operator calculation, the weights are sliced and inherited using the following formula to achieve an equivalent output result with partially invalid results.

[0158] Assuming that the original matrix multiplication calculation logic is input·weight=output, the algorithm described in this embodiment performs a binary gated value masked dot product on the head dimension of the multi-head attention output output, that is, a certain dimension in the output is set to 0, and the final output is recorded as pruned_output. By masking the weight, input·masked_weight=pruned_output, the final output pruned_output can be calculated, as shown in formulas (7) and (8). Since there are a large number of elements with a value of 0 in pruned_output, the subsequent calculation results are also 0, that is, some elements of pruned_output are invalid in the forward transmission. Therefore, it can be considered to slice and inherit the weight weight to obtain pruned_weight, and then perform matrix operations with the input to obtain a result equivalent to pruned_output, as shown in formula (9).

[0159]

[0160]

[0161]

[0162] Similarly, by implementing the aforementioned parameter slicing and inheritance logic into each parameterized operator in the multi-head attention module and performing pruning within the pre-pruned model, we can trim the actual parameters of the target pre-trained model, reducing the number of model parameters and lowering the deployment requirements and threshold. Therefore, based on the threshold network distribution of the pre-pruned model, an automated model pruning tool is developed to produce the final pruned model.

[0163] S600: Obtain a target publishing model according to the target pruning model.

[0164] Finally, after completing the above steps, each computing center locally generates the target pruned model with the final parameter reduction. The model accuracy is evaluated on the validation set of the target dataset to obtain the target release model, and the model file is prepared for release and deployment.

[0165] In summary, this embodiment provides a large model online pruning method with cross-domain data heterogeneity, which inputs a reference image into a feature extraction network, obtains a target pre-trained model and a target data set; pre-prunes the target pre-trained model according to the target data set to obtain a pre-pruned model; sends the parameters of the pre-pruned model to a cloud server for processing to obtain target fusion parameters, loads the target fusion parameters into the pre-pruned model to obtain a target fusion model, and updates the target pre-trained model to the target fusion model; re-executes the step of pre-pruning the target pre-trained model according to the target data set until a preset number of rounds are completed, and uses the target fusion model as the target pre-pruned model; performs structural pruning on the target pre-pruned model to obtain a target pruned model; and obtains a target release model based on the target pruned model. The present invention can perform online pruning on large models in a cross-domain network environment, reduce model parameters, and lower the threshold for hardware requirements for large model deployment.

[0166] It should be understood that, although the steps in the flowcharts provided in the accompanying drawings of the present invention are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps may be performed in other orders. Moreover, at least a portion of the steps in the flowcharts may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be performed in rotation or alternation with other steps or at least a portion of sub-steps or stages of other steps.

[0167] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0168] Example 2

[0169] Based on the above embodiments, the present invention also provides a large model online pruning device with cross-domain data heterogeneity. Figure 4 As shown, the large model online pruning device with cross-domain data heterogeneity includes:

[0170] A data acquisition module, which is used to acquire a target pre-trained model and a target data set, as described in the first embodiment;

[0171] A pre-pruned model acquisition module, the pre-pruned model acquisition module is used to pre-prune the target pre-trained model according to the target data set to obtain a pre-pruned model, as described in the first embodiment;

[0172] A fusion model acquisition module is used to send the pre-pruned model parameters to a cloud server for processing to obtain target fusion parameters, load the target fusion parameters into the pre-pruned model to obtain a target fusion model, and update the target pre-trained model to the target fusion model, as described in Example 1;

[0173] An updating module, configured to re-execute the step of pre-pruning the target pre-trained model according to the target dataset until a preset number of rounds are completed, and then use the target fusion model as the target pre-pruned model, as described in the first embodiment;

[0174] a pruning model acquisition module, configured to perform structural pruning on the target pre-pruning model to obtain a target pruning model, as specifically described in the first embodiment;

[0175] The publishing model acquisition module is used to obtain the target publishing model according to the target pruning model, as described in the first embodiment.

[0176] Example 3

[0177] Based on the above embodiments, the present invention also provides a terminal, such as Figure 5 As shown, the terminal includes a processor 10 and a memory 20. Figure 5 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0178] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or is to be output. In one embodiment, a large model online pruning program 30 with cross-domain data heterogeneity is stored on the memory 20, and the large model online pruning program 30 with cross-domain data heterogeneity can be executed by the processor 10, thereby realizing the large model online pruning method with cross-domain data heterogeneity in the present application.

[0179] In some embodiments, the processor 10 can be a central processing unit (CPU), a microprocessor or other chip, used to run the program code or process data stored in the memory 20, such as executing the online pruning method for large models with cross-domain data heterogeneity.

[0180] In one embodiment, when the processor 10 executes the large model online pruning program 30 for cross-domain data heterogeneity in the memory 20, the following steps are implemented:

[0181] Obtain the target pre-trained model and target dataset;

[0182] Pre-pruning the target pre-trained model according to the target data set to obtain a pre-pruned model;

[0183] Sending the parameters of the pre-pruned model to a cloud server for processing to obtain target fusion parameters, loading the target fusion parameters into the pre-pruned model to obtain a target fusion model, and updating the target pre-trained model to the target fusion model;

[0184] Re-performing the step of pre-pruning the target pre-trained model according to the target data set until a preset number of rounds are completed, and then using the target fusion model as the target pre-pruned model;

[0185] Performing structural pruning on the target pre-pruned model to obtain a target pruned model;

[0186] A target publishing model is obtained according to the target pruning model.

[0187] The target pre-training model is a transformer model.

[0188] Before obtaining the target pre-trained model and the target dataset, the method further includes:

[0189] Acquire an original data set, and preprocess the original data to obtain the target data set;

[0190] The target dataset template is consistent with the preset template.

[0191] The pre-pruning of the target pre-trained model according to the target data set includes:

[0192] Obtain the multi-head attention module in the target pre-trained model;

[0193] Setting a corresponding gating network according to each head in the multi-head attention module;

[0194] Perform a dot product between the output of the multi-head attention module corresponding to each training data in the target data set and the gated network to obtain the target attention output;

[0195] A pre-pruning training loss is obtained according to the gated sparsification function and the target attention output, and the target pre-training model is updated according to the optimization target.

[0196] Wherein, the gated sparsification function is:

[0197]

[0198] Among them, the number of layers of the multi-head attention module in the target pre-training model is l, G i is the gating network corresponding to the multi-layer attention module in the i-th layer, i∈[0,l).

[0199] The pre-pruning training loss is:

[0200]

[0201] Among them, θ is the original transformer model parameter, are the parameters of the gating network part, is the cross entropy loss function of the target pre-trained model, is the gated sparsity loss function, and λ is a hyperparameter.

[0202] Among them, the target fusion parameter is obtained by the cloud server calculating the weighted average of the non-gated network part model parameters in the parameters of the pre-pruned models of different computing centers, and performing an extreme value selection fusion strategy on the gated network part in the pre-pruned model parameters of different computing centers.

[0203] The performing structural pruning on the target pre-pruning model includes:

[0204] Performing binarization approximation processing on the gating value in the gating network to obtain a binarized gating value, wherein the binarized gating value is 0 or 1;

[0205] Structural pruning is performed on the target pre-pruned model according to the binarized gating value to obtain a target pruning model.

[0206] The performing binarization approximation processing on the gated value in the gated network to obtain the binarized gated value includes:

[0207] Using the first formula to obtain the binarized gating value;

[0208] The first formula is

[0209]

[0210]

[0211]

[0212] Wherein, z is the binarized gate value, u is the random noise that obeys the uniform distribution of U(0,1), g jFor the i-th gating network G i The jth element of , β is a hyperparameter, s is the distribution of the gating network mapped to the (0,1) interval after adding noise, is the intermediate parameter, γ and ζ are parameters, γ<0,ζ>1.

[0213] The step of performing structural pruning on the target pre-pruning model according to the binary gating value includes:

[0214] The attention heads in the multi-head attention module corresponding to the binary gating value of 0 are pruned.

[0215] Example 4

[0216] The present invention also provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the online pruning method for large models with cross-domain data heterogeneity as described above.

[0217] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A large-scale online pruning method for cross-domain data heterogeneity, characterized by: The online pruning method for a large model with heterogeneous cross-domain data includes: Obtain the target pre-trained model and target dataset; Pre-pruning the target pre-trained model according to the target data set to obtain a pre-pruned model; Sending the parameters of the pre-pruned model to a cloud server for processing to obtain target fusion parameters, loading the target fusion parameters into the pre-pruned model to obtain a target fusion model, and updating the target pre-trained model to the target fusion model; Re-performing the step of pre-pruning the target pre-trained model according to the target data set until a preset number of rounds are completed, and then using the target fusion model as the target pre-pruned model; Performing structural pruning on the target pre-pruned model to obtain a target pruned model; Obtaining a target release model according to the target pruning model; The pre-pruning of the target pre-trained model according to the target data set includes: Obtain the multi-head attention module in the target pre-trained model; Setting a corresponding gating network according to each head in the multi-head attention module; Perform a dot product between the output of the multi-head attention module corresponding to each training data in the target data set and the gated network to obtain the target attention output; Obtaining a pre-pruning training loss based on a gated sparsification function and the target attention output, and updating the target pre-trained model based on an optimization objective; The gated sparsification function is: ； Among them, the number of layers of the multi-head attention module in the target pre-training model is l , is the gating network corresponding to the i-th layer multi-layer attention module, .

2. The online pruning method for large models with heterogeneous cross-domain data according to claim 1 is characterized in that: The target pre-training model is a transformer model.

3. The online pruning method for large models with heterogeneous cross-domain data according to claim 1 is characterized in that: Before obtaining the target pre-trained model and the target dataset, the method further includes: Acquire an original data set, and preprocess the original data to obtain the target data set; The target dataset template is consistent with the preset template.

4. The online pruning method for large models with heterogeneous cross-domain data according to claim 1 is characterized in that: The pre-pruning training loss is: ； in, are the original transformer model parameters, are the parameters of the gating network part, is the cross entropy loss function of the target pre-trained model, is the gated sparsity loss function, is a hyperparameter.

5. The online pruning method for large models with heterogeneous cross-domain data according to claim 1 is characterized in that: The target fusion parameters are obtained by the cloud server calculating the weighted average of the non-gated network part model parameters in the parameters of the pre-pruned models of different computing centers, and performing an extreme value selection fusion strategy on the gated network part in the pre-pruned model parameters of different computing centers.

6. The online pruning method for large models with heterogeneous cross-domain data according to claim 1 is characterized in that: The performing structural pruning on the target pre-pruning model includes: Performing binarization approximation processing on the gating value in the gating network to obtain a binarized gating value, wherein the binarized gating value is 0 or 1; Structural pruning is performed on the target pre-pruned model according to the binarized gating value to obtain a target pruning model.

7. The online pruning method for large models with heterogeneous cross-domain data according to claim 6 is characterized in that: The performing binarization approximation processing on the gating value in the gating network to obtain the binarized gating value includes: Using the first formula to obtain the binarized gating value; The first formula is: ； ； ； Wherein, z is the binarization gate value, To obey Uniformly distributed random noise, For the i-th gating network The jth element of is a hyperparameter, s is the distribution of the gating network mapped to the (0,1) interval after adding noise, is the intermediate parameter, 、 is the parameter, .

8. The online pruning method for large models with heterogeneous cross-domain data according to claim 6 is characterized in that: The performing structural pruning on the target pre-pruning model according to the binarized gating value includes: The attention heads in the multi-head attention module corresponding to the binary gating value of 0 are pruned.

9. A large model online pruning device for cross-domain data heterogeneity, characterized by: include: A data acquisition module, wherein the acquisition module is used to acquire a target pre-training model and a target data set; A pre-pruned model acquisition module, configured to pre-prune the target pre-trained model according to the target data set to obtain a pre-pruned model; A fusion model acquisition module is used to send the pre-pruned model parameters to the cloud server for processing to obtain target fusion parameters, load the target fusion parameters into the pre-pruned model to obtain a target fusion model, and update the target pre-trained model to the target fusion model; An updating module, configured to re-execute the step of pre-pruning the target pre-trained model according to the target data set until a preset number of rounds are completed, and then use the target fusion model as the target pre-pruned model; A pruning model acquisition module, configured to perform structural pruning on the target pre-pruning model to obtain a target pruning model; A publishing model acquisition module, configured to obtain a target publishing model based on the target pruning model; The pre-pruning of the target pre-trained model according to the target data set includes: Obtain the multi-head attention module in the target pre-trained model; Setting a corresponding gating network according to each head in the multi-head attention module; Perform a dot product between the output of the multi-head attention module corresponding to each training data in the target data set and the gated network to obtain the target attention output; Obtaining a pre-pruning training loss based on a gated sparsification function and the target attention output, and updating the target pre-trained model based on an optimization objective; The gated sparsification function is: ； Among them, the number of layers of the multi-head attention module in the target pre-training model is l , is the gating network corresponding to the i-th layer multi-layer attention module, .

10. A terminal, characterized in that: The terminal includes: a processor, a computer-readable storage medium communicatively connected to the processor, the computer-readable storage medium being suitable for storing a plurality of instructions, and the processor being suitable for calling the instructions in the computer-readable storage medium to execute the steps of the online pruning method for a large model with cross-domain data heterogeneity as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the online pruning method for a large model with cross-domain data heterogeneity as described in any one of claims 1-8.

Images