A model pruning method and device and a storage medium

Abstract

The embodiment of the application discloses a model pruning method and device and a storage medium. The method comprises the following steps: in the encoder of an original transformation model, full connection layers with an association relationship are divided into a group of full connection layers, and a corresponding mask layer is preset for each group of full connection layers to generate a to-be-trained transformation model; wherein each mask layer respectively acts on each full connection layer in the corresponding group of full connection layers, and is used for pruning the output channels of each full connection layer in the corresponding group of full connection layers; the to-be-trained transformation model is trained by using the original transformation model and training samples to obtain a pruning transformation model corresponding to the original transformation model.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application relates to the technical field of artificial intelligence, in particular to a model pruning method and device and storage medium. BACKGROUND

[0002] At present, the transformer model pruning method can be mainly divided into two kinds: the first pruning method is to learn dimension reduction on the vectors transmitted between the encoders of the transformer model; the second pruning method is to regard the transformer model as a stack of independent fully connected layers, and decompose the pruning of the transformer model into the pruning of the fully connected layers.

[0003] However, for the first pruning method, the transformer model obtained by pruning often damages its generalization for downstream tasks, such as object detection, instance segmentation and the like. In addition, this method is difficult to apply to natural language processing tasks, and the application range of this pruning method is limited. For the second pruning method, simply regarding the transformer model as a stack of independent fully connected layers for pruning has limitations in precision and pruning rate. SUMMARY

[0004] The embodiment of the present application provides a model pruning method, device and storage medium, which not only improves the pruning rate and precision, but also guarantees the generalization of the transformer model obtained by pruning for downstream tasks.

[0005] The technical scheme of the embodiment of the present application is implemented as follows:

[0006] The embodiment of the present application provides a model pruning method, comprising:

[0007] In the encoder of the original transformer model, the fully connected layers with an association relationship are divided into a group of fully connected layers, and a corresponding mask layer is preset for each group of fully connected layers to generate a to-be-trained transformer model; wherein each mask layer respectively acts on each fully connected layer in the corresponding group of fully connected layers, and is used for pruning the output channels of each fully connected layer in the corresponding group of fully connected layers.

[0008] The to-be-trained transformer model is trained by using the original transformer model and training samples to obtain a pruned transformer model corresponding to the original transformer model.

[0009] In the above method, the encoder of the original transformation model is composed of a sequentially connected attention network and a feedforward network, the attention network includes a query full connection layer, a key full connection layer, a value full connection layer and a projection full connection layer, and the feedforward network includes a first full connection layer and a second full connection layer connected with the first full connection layer, wherein the output channel numbers of the query full connection layer, the key full connection layer and the value full connection layer are consistent, the output channel numbers of the projection full connection layer and the second full connection layer are consistent, and the full connection layers having an association relationship in the encoder of the original transformation model are divided into a group of full connection layers, including:

[0010] The query full connection layer, the key full connection layer and the value full connection layer are divided into a group of full connection layers;

[0011] The first full connection layer is separately divided into a group of full connection layers;

[0012] The projection full connection layer and the second full connection layer are divided into a group of full connection layers.

[0013] In the above method, the training of the to-be-trained transformation model by using the original transformation model and the training sample includes:

[0014] Processing the training sample by using the original transformation model and the to-be-trained transformation model respectively to obtain a first processing result corresponding to the original transformation model and a second processing result corresponding to the to-be-trained transformation model;

[0015] Updating the mask layer corresponding to each group of full connection layers in the encoder of the to-be-trained transformation model based on a difference loss function between the first processing result and the second processing result.

[0016] In the above method, the difference loss function at least includes one or more of a mean square error loss function, a relative entropy loss function and an adversarial loss function.

[0017] In the above method, the difference loss function includes a relative entropy loss function, the first processing result and the second processing result are both probability distributions corresponding to the training sample, and the relative entropy loss function is:

[0018]

[0019] Wherein, L KL is the relative entropy loss function, n is the number of probability values contained in the first processing result and the second processing result, P i is the i-th probability value in the first processing result, Q i is the i-th probability value in the second processing result, and Pi Q i The probability values belong to the same dimension.

[0020] In the method, the difference loss function comprises an adversarial loss function, and before the encoder of the to-be-trained transformation model is updated based on the difference loss function between the first processing result and the second processing result, the method further comprises:

[0021] The first processing result and the second processing result are input into a decision maker for reliability decision, and the adversarial loss function is generated according to a decision result.

[0022] In the method, the difference loss function comprises a mean square error loss function, a relative entropy loss function and an adversarial loss function, and the encoder of the to-be-trained transformation model is updated based on the difference loss function between the first processing result and the second processing result, comprising:

[0023] The sum of the mean square error loss function, the relative entropy loss function and the adversarial loss function is calculated to obtain a total loss function;

[0024] The encoder of the to-be-trained transformation model is updated based on the total loss function.

[0025] Embodiments of the present application provide a model pruning device, comprising:

[0026] A generation module is configured to divide full connection layers with an association relationship into a group of full connection layers in an encoder of an original transformation model, and preset a corresponding mask layer for each group of full connection layers to generate a to-be-trained transformation model; wherein each mask layer respectively acts on each full connection layer in a corresponding group of full connection layers, and is used for pruning an output channel of each full connection layer in a corresponding group of full connection layers.

[0027] A training module is configured to train the to-be-trained transformation model by using the original transformation model and training samples to obtain a pruning transformation model corresponding to the original transformation model.

[0028] Embodiments of the present application provide a model pruning device, comprising: a processor, a memory and a communication bus;

[0029] The communication bus is configured to realize the communication connection between the processor and the memory.

[0030] The processor is configured to execute one or more computer programs stored in the memory to realize the above-mentioned model pruning method.

[0031] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the model pruning method.

[0032] The embodiment of the present application provides a model pruning method, device and storage medium, the method comprising: in the encoder of the original transformation model, the fully connected layers with an association relationship are divided into a group of fully connected layers, and a corresponding mask layer is preset for each group of fully connected layers to generate a to-be-trained transformation model; wherein each mask layer respectively acts on each fully connected layer in the corresponding group of fully connected layers, and is used for pruning the output channels of each fully connected layer in the corresponding group of fully connected layers; using the original transformation model and training samples, the to-be-trained transformation model is trained to obtain a pruning transformation model corresponding to the original transformation model. The technical scheme provided by the embodiment of the present application designs a shareable mask layer for the fully connected layers with an association relationship in the encoder according to the intrinsic property of the encoder of the transformation model, so as to realize the pruning of the fully connected layers, thereby not only improving the pruning rate and the accuracy, but also guaranteeing the generalization of the transformation model obtained by pruning for downstream tasks. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A flowchart of a model pruning method provided by the embodiment of the present application is shown in the figure.

[0034] Figure 2 A structure diagram of an exemplary original transformation model provided by the embodiment of the present application is shown in the figure.

[0035] Figure 3 A structure diagram of an exemplary encoder provided by the embodiment of the present application is shown in the figure.

[0036] Figure 4 An exemplary fully connected layer pruning diagram provided by the embodiment of the present application is shown in the figure.

[0037] Figure 5 A training process diagram of an exemplary to-be-trained transformation model provided by the embodiment of the present application is shown in the figure.

[0038] Figure 6 A structure diagram of a model pruning device provided by the embodiment of the present application is shown in the figure. Figure 1 ;

[0039] Figure 7 A structure diagram of a model pruning device provided by the embodiment of the present application is shown in the figure. Figure 2 . DETAILED DESCRIPTION

[0040] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0041] The technical solutions of the present application and how the technical solutions solve the above technical problems will be described in detail below through embodiments and in combination with the drawings. The following embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments.

[0042] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0043] The model pruning method provided by the embodiments of the present application is implemented by a model pruning device. The specific type of the model pruning device is not limited in the embodiments of the present application, and it can be any electronic device.

[0044] Figure 1 A flowchart of a model pruning method provided by the embodiments of the present application is shown in FIG. 1. As shown in FIG. 1, in the embodiments of the present application, the model pruning method mainly includes the following steps: Figure 1

[0045] S101, in the encoder of the original transformation model, the fully connected layers having a correlation relationship are divided into a group of fully connected layers, and a corresponding mask layer is preset for each group of fully connected layers to generate a to-be-trained transformation model; wherein each mask layer respectively acts on each fully connected layer in the corresponding group of fully connected layers, and is used to prune the output channels of each fully connected layer in the corresponding group of fully connected layers.

[0046] In the embodiments of the present application, the model pruning device can divide the fully connected layers having a correlation relationship into a group of fully connected layers in the encoder of the original transformation model, and preset a corresponding mask layer for each group of fully connected layers to generate a to-be-trained transformation model; each mask layer respectively acts on each fully connected layer in the corresponding group of fully connected layers, and is used to prune the output channels of each fully connected layer in the corresponding group of fully connected layers.

[0047] It should be noted that in the embodiments of the present application, the original transformation model can be a pre-generated or acquired transformation model, which is composed of multiple cascaded encoders, as shown in FIG. 2. The specific original transformation model and the number of encoders of the original transformation model can be set according to actual needs and application scenarios, and the embodiments of the present application are not limited. The model pruning device can perform grouping and mask layer setting operations for each encoder respectively. Figure 2

[0048] ​​Specifically, in the embodiments of the present application, the encoder of the original transformation model is composed of a sequentially connected attention network and a feedforward network, the attention network includes a query full connection layer, a key full connection layer, a value full connection layer and a projection full connection layer, and the feedforward network includes a first full connection layer and a second full connection layer connected with the first full connection layer, wherein the output channel numbers of the query full connection layer, the key full connection layer and the value full connection layer are required to be consistent, the output channel numbers of the projection full connection layer and the second full connection layer are required to be consistent, and in the encoder of the original transformation model, the full connection layers having an association relationship are divided into a group of full connection layers, including: the query full connection layer, the key full connection layer and the value full connection layer are divided into a group of full connection layers; the first full connection layer is separately divided into a group of full connection layers; and the projection full connection layer and the second full connection layer are divided into a group of full connection layers.

[0049] Figure 3 An exemplary encoder structure diagram is provided for the embodiments of the present application. As shown in the figure, the encoder is composed of a sequentially connected attention network and a feedforward network. Figure 3 As shown in the figure, the encoder is composed of a sequentially connected attention network and a feedforward network. q k v proj , and a projection full connection layer FC q , a key full connection layer FC k and a value full connection layer FC v , the output of the multi-head self-attention layer is input into the projection full connection layer FC proj , in addition, the input data of the attention network needs to be normalized. The feedforward network includes a first full connection layer FC1 and a second full connection layer FC2, the input data of the first full connection layer FC1 needs to be normalized, and the output data needs to be activated. As shown in the figure, the input channel number and the output channel number of each full connection layer are marked after the full connection layer. For example, DxD, the first D is the output channel number of the full connection layer, and the second D is the input channel number of the full connection layer. Figure 3

[0050] The grouping of full connection layers is described in detail below based on the encoder structure shown in the figure. Figure 3

[0051] In the embodiments of the present application, for the multi-head self-attention layer shown in the figure, the calculation method is as shown in the following formula: Figure 3

[0052] ​​​​​​

[0053] Here, Attention(Q,K,V) represents the output of the multi-head self-attention layer, and the three matrices Q, K, and V are respectively generated by the query fully connected layer FC. q Fully Connected Layer (FC) k Sum of fully connected layers (FC) v The three fully connected layers correspond to the same output channel dimension, which is D.

[0054] It should be noted that the multiplication of Q, K, and V in formula (1) is a matrix multiplication, therefore, for the query fully connected layer FC... q Fully Connected Layer (FC) k Sum of fully connected layers (FC) v It's important to note that the number of output channels should remain consistent after pruning; otherwise, it will affect the calculation of the self-attention mechanism. For these reasons, the query fully connected layer needs to be FC... q Fully Connected Layer (FC) k Sum of fully connected layers (FC) v They are grouped together.

[0055] In the embodiments of this application, such as Figure 3 As shown, the first fully connected layer FC1 is not associated with other fully connected layers. Therefore, the first fully connected layer FC1 is divided into a separate group.

[0056] In the embodiments of this application, such as Figure 3 As shown, both the attention network and the feedforward network have a short connection, which introduces some dependencies: querying the fully connected layer (FC). q Fully Connected Layer (FC) k Sum of fully connected layers (FC) v The number of input channels should be the same as the number of fully connected projection layers (FC). proj The number of output channels is consistent; the projected fully connected layer FC proj The number of output channels should be the same as the number of output channels of the second fully connected layer FC2. The original transform model is typically composed of multiple encoders connected in series; the number of output channels of the second fully connected layer FC2 of the current encoder is the same as the number of output channels of the query fully connected layer FC2 of the next encoder. q Fully Connected Layer (FC) k Sum of fully connected layers (FC) v The number of input channels. In summary, the projected fully connected layer (FC) proj The number of output channels of the second fully connected layer FC2 should be the same, therefore, they can be grouped together.

[0057] In the embodiments of the present application, the model pruning device can preset one mask layer for each group of full connection layers after dividing the three groups of full connection layers, and each mask layer is used to prune the output channels of each full connection layer in the corresponding group of full connection layers. Each full connection layer in the same group of full connection layers is actually pruned by the same mask layer.

[0058] It should be noted that in the embodiments of the present application, as shown in Figure 4 There can be multiple output channels in one full connection layer, and the different pattern filled squares in the full connection layer in the figure represent different output channels. The model pruning device can apply the mask layer corresponding to the full connection layer group to which the full connection layer belongs to the full connection layer. The masks in the mask layer can be trained, and they are initialized to 1. After training, most of the masks in the mask layer will converge to 0. If a mask in the mask layer is 0, it means that the corresponding output channel in the full connection layer is pruned.

[0059] S102, training the to-be-trained transformation model using the original transformation model and the training sample to obtain a pruning transformation model corresponding to the original transformation model.

[0060] In the embodiments of the present application, after obtaining the to-be-trained transformation model, the model pruning device can train the to-be-trained transformation model using the original transformation model and the training sample to obtain a pruning transformation model corresponding to the original transformation model.

[0061] Specifically, in the embodiments of the present application, the model pruning device trains the to-be-trained transformation model using the original transformation model and the training sample, including: processing the training sample using the original transformation model and the to-be-trained transformation model respectively to obtain a first processing result corresponding to the original transformation model and a second processing result corresponding to the to-be-trained transformation model; and updating the mask layer corresponding to each group of full connection layers in the encoder of the to-be-trained transformation model based on a difference loss function between the first processing result and the second processing result.

[0062] It should be noted that in the embodiments of the present application, the model pruning device can obtain the training sample in advance. The training sample can be a multimedia file such as an image, a video, or a voice. Correspondingly, the original transformation model and the to-be-trained transformation model are transformation models that can process multimedia files such as images, videos, and voices. The specific training sample is not limited in the embodiments of the present application.

[0063] It can be understood that in the embodiments of the present application, the model pruning device can input the training sample into the original transformation model and the to-be-trained transformation model for processing, so as to output the first processing result from the original transformation model and the second processing result from the to-be-trained transformation model.

[0064] It should be noted that in the embodiments of the present application, the model pruning device updates the mask layer corresponding to each group of fully connected layers in the encoder of the to-be-trained transformation model based on the difference loss function between the first processing result and the second processing result. Of course, other network parameters in the to-be-trained transformation model can also be updated based on the difference loss function, and the embodiments of the present application are not limited thereto.

[0065] It can be understood that in the embodiments of the present application, in step S101, the model pruning device only preconfigures the corresponding mask layer for each group of fully connected layers, and the masks in the mask layer are directly preconfigured according to experience or are all set to 1. Therefore, the network effect of the to-be-trained transformation model cannot necessarily approach the network effect of the original transformation model, and thus the mask layer of the to-be-trained transformation model needs to be updated according to the difference between the processing results of the original transformation model and the to-be-trained transformation model for the training samples to meet the actual model effect requirement.

[0066] It should be noted that in the embodiments of the present application, the model pruning device can calculate the difference loss function between the processing results of the two transformation models, and through the difference loss function, the network effect difference of the to-be-trained transformation model relative to the original transformation model can be measured. Thus, the update of the mask layer corresponding to each group of fully connected layers in the encoder of the to-be-trained transformation model can be performed based on the difference loss function.

[0067] It should be noted that in the embodiments of the present application, the difference loss function between the first processing result and the second processing result at least includes one or more of a mean square error loss function, a relative entropy loss function, and an adversarial loss function, and the embodiments of the present application are not limited thereto.

[0068] Specifically, in the embodiments of the present application, in the case where the difference loss function between the first processing result and the second processing result includes the relative entropy loss function, the first processing result and the second processing result are both the probability distribution corresponding to the training sample, and the relative entropy loss function L KL is:

[0069]

[0070] wherein n is the number of probability values contained in the first processing result and the second processing result, P i is the i-th probability value in the first processing result, Q i is the i-th probability value in the second processing result, P i and Q i are probability values in the same dimension.

[0071] Specifically, in the embodiments of the present application, the difference loss function between the first processing result and the second processing result includes an adversarial loss function, and the model pruning device further performs the following steps before updating the mask layer corresponding to each group of full connection layers in the encoder of the to-be-trained transformation model based on the difference between the first processing result and the second processing result in the loss function: inputting the first processing result and the second processing result into a decision maker for reliability decision, and generating an adversarial loss function according to the decision result.

[0072] It should be noted that in the embodiments of the present application, the model pruning device can input the first processing result and the second processing result into the decision maker for reliability decision, so that the corresponding decision result can be obtained, and the difference between the two processing result decision results can be used to generate the adversarial loss function.

[0073] It should be noted that in the embodiments of the present application, the model pruning device can directly use the existing mean square error loss function formula to calculate the mean square error loss function of the first processing result and the second processing result, which will not be described here.

[0074] Specifically, in the embodiments of the present application, when the difference loss function includes the mean square error loss function, the relative entropy loss function and the adversarial loss function, the model pruning device updates the mask layer corresponding to each group of full connection layers in the encoder of the to-be-trained transformation model based on the difference loss function between the first processing result and the second processing result, including: calculating the sum of the mean square error loss function, the relative entropy loss function and the adversarial loss function to obtain a total loss function; and updating the mask layer corresponding to each group of full connection layers in the encoder of the to-be-trained transformation model based on the total loss function.

[0075] It should be noted that in the embodiments of the present application, the total loss function L is as follows:

[0076] L = L KL + L MSE + L GAN (3)

[0077] Wherein, L KL is the relative entropy loss function, L MSE is the mean square error loss function, and L GAN is the adversarial loss function.

[0078] It should be noted that in the embodiments of the present application, the model pruning device can not only update the mask layer corresponding to each group of full connection layers in the encoder of the to-be-trained transformation model based on the difference loss function between the first processing result and the second processing result, but also update other network parameters, so as to finally obtain the pruned transformation model.

[0079] Figure 5An exemplary training process diagram of a to-be-trained transformation model is provided in the embodiments of the present application. As shown in Figure 5 , the original transformation model is a network to be pruned, and the network parameters thereof are not updated through back propagation. The to-be-trained transformation model is based on the original transformation model, and the transformation model after adding the mask layer is obtained by performing step S101. The network parameters of the transformation model after adding the mask layer inherit the network parameters of the original transformation model except that the mask in the mask layer is initialized as 1. In addition, the mask layer participates in the update of the back propagation, and other parameters also participate in the update of the back propagation. The difference loss function includes a mean square error loss function L MSE , a relative entropy loss function L KL , and an adversarial loss function L GAN .

[0080] It can be understood that, in the embodiments of the present application, as shown in Figure 5 , the difference loss function includes not only the mean square error loss function L MSE and the adversarial loss function L GAN , but also the relative entropy loss function L KL , so as to further constrain the entire pruning training process, so that the final pruned transformation model is more compact and maintains the accuracy.

[0081] It can be understood that, in the embodiments of the present application, for the encoder of the original transformation model, according to its intrinsic properties, the correlation between the internal fully connected layers is considered, and the fully connected layers are divided into three parts, each part is trained using a shared mask layer, so as to not only improve the pruning rate, but also improve the pruning accuracy. In addition, the constraint of the relative entropy loss function is introduced into the training process, so that the pruned transformation model obtained by pruning is more compact.

[0082] The embodiments of the present application provide a model pruning method, which comprises: in an encoder of an original transformation model, dividing fully connected layers having a correlation relationship into a group of fully connected layers, and pre-setting a corresponding mask layer for each group of fully connected layers to generate a to-be-trained transformation model; wherein each mask layer respectively acts on each fully connected layer in a corresponding group of fully connected layers, and is used for pruning the output channels of each fully connected layer in the corresponding group of fully connected layers; and using the original transformation model and training samples to train the to-be-trained transformation model to obtain a pruned transformation model corresponding to the original transformation model. The model pruning method provided by the embodiments of the present application designs a shareable mask layer for the fully connected layers having a correlation relationship in the encoder of the transformation model according to the intrinsic properties of the encoder, so as to realize the pruning of the fully connected layers, thereby not only improving the pruning rate and accuracy, but also ensuring the generalization of the pruned transformation model to downstream tasks.

[0083] The embodiments of the present application provide a model pruning device.Figure 6 A structure diagram of a model pruning device provided in an embodiment of the present application Figure 1 As shown in the figure, in the embodiment of the present application, the model pruning device comprises: Figure 6

[0084] The generating module 601 is configured to divide the full connection layers with an association relationship into a group of full connection layers in the encoder of the original transformation model, and preset a corresponding mask layer for each group of full connection layers to generate a to-be-trained transformation model; wherein each mask layer respectively acts on each full connection layer in the corresponding group of full connection layers, and is used to prune the output channels of each full connection layer in the corresponding group of full connection layers

[0085] The training module 602 is configured to train the to-be-trained transformation model by using the original transformation model and training samples to obtain a pruned transformation model corresponding to the original transformation model.

[0086] In an embodiment of the present application, the encoder of the original transformation model is composed of a sequentially connected attention network and a feedforward network, the attention network comprises a query full connection layer, a key full connection layer, a value full connection layer and a projection full connection layer, and the feedforward network comprises a first full connection layer and a second full connection layer connected with the first full connection layer, wherein the output channel numbers of the query full connection layer, the key full connection layer and the value full connection layer are required to be consistent, and the output channel numbers of the projection full connection layer and the second full connection layer are required to be consistent.

[0087] The generating module 601 is further configured to divide the query full connection layer, the key full connection layer and the value full connection layer into a group of full connection layers; divide the first full connection layer into a group of full connection layers alone; and divide the projection full connection layer and the second full connection layer into a group of full connection layers.

[0088] In an embodiment of the present application, the training module 602 is further configured to process training samples by using the original transformation model and the to-be-trained transformation model respectively to obtain a first processing result corresponding to the original transformation model and a second processing result corresponding to the to-be-trained transformation model; and update the mask layer corresponding to each group of full connection layers in the encoder of the to-be-trained transformation model based on a difference loss function between the first processing result and the second processing result.

[0089] In an embodiment of the present application, the difference loss function at least includes one or more of a mean square error loss function, a relative entropy loss function and an adversarial loss function.

[0090] ​In an embodiment of the present application, the difference loss function comprises a relative entropy loss function, the first processing result and the second processing result are both probability distributions corresponding to the training sample, and the relative entropy loss function is:

[0091]

[0092] wherein L KL is the relative entropy loss function, n is the number of probability values contained in each of the first processing result and the second processing result, P i is the i-th probability value in the first processing result, Q i is the i-th probability value in the second processing result, P i and Q i are probability values in the same dimension.

[0093] In an embodiment of the present application, the difference loss function comprises an adversarial loss function, and the training module 602 is further configured to input the first processing result and the second processing result into a decider to perform reliability judgment, and generate the adversarial loss function according to a judgment result.

[0094] In an embodiment of the present application, the difference loss function comprises a mean square error loss function, a relative entropy loss function and an adversarial loss function, and the training module 602 is further configured to calculate a sum of the mean square error loss function, the relative entropy loss function and the adversarial loss function to obtain a total loss function, and update a mask layer corresponding to each group of fully connected layers in the encoder of the to-be-trained transformation model based on the total loss function.

[0095] Figure 7 FIG. 7 shows a structure of a model pruning device provided in an embodiment of the present application Figure 2 . As shown in FIG. 7, in an embodiment of the present application, the model pruning device comprises a processor 701, a memory 702 and a communication bus 703. Figure 7

[0096] The communication bus 703 is configured to realize a communication connection between the processor 701 and the memory 702.

[0097] The processor 701 is configured to execute one or more computer programs stored in the memory 702 to realize the model pruning method described above.

[0098] ​The embodiment of the present application provides a model pruning device, in the encoder of an original transformation model, full connection layers with an association relationship are divided into a group of full connection layers, and a corresponding mask layer is preset for each group of full connection layers to generate a to-be-trained transformation model; wherein each mask layer respectively acts on each full connection layer in a corresponding group of full connection layers, and is used for pruning the output channel of each full connection layer in the corresponding group of full connection layers; the original transformation model and the training sample are used to train the to-be-trained transformation model, and a pruning transformation model corresponding to the original transformation model is obtained. The model pruning device provided by the embodiment of the present application is designed according to the internal nature of the encoder of the transformation model, and a shareable mask layer is designed for the full connection layers with an association relationship in the encoder, so that the pruning of the full connection layers is realized, thereby not only improving the pruning rate and the accuracy, but also ensuring the generalization of the transformation model obtained by pruning for downstream tasks.

[0099] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the above-mentioned model pruning method. The computer readable storage medium can be a volatile memory (volatile memory), such as random access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as read-only memory (Read-Only Memory, ROM), flash memory, hard disk (Hard Disk Drive, HDD) or solid state disk (Solid-State Drive, SSD); it can also be a respective device including one or any combination of the above-mentioned memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant.

[0100] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a hardware embodiment, a software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage and optical storage) containing computer usable program code.

[0101] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 one or more functions specified in the flowchart or flowsheet block or blocks. Figure 1

[0102] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart or flowsheet block or blocks. Figure 1 one or more functions specified in the flowchart or flowsheet block or blocks. Figure 1

[0103] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure 1 one or more functions specified in the flowchart or flowsheet block or blocks. Figure 1 Figure 1

[0104] The above generally describes only some embodiments of the present application, and the protection scope of the present application is not limited thereto. Any changes or replacements easily conceived by those skilled in the art within the technical scope disclosed by the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.​​​

Claims

1. A model pruning method, characterized in that, include: In the encoder of the original transformation model, the fully connected layers with correlation are divided into a group of fully connected layers, and a corresponding mask layer is preset for each group of fully connected layers to generate the transformation model to be trained; wherein, each mask layer acts on each fully connected layer in the corresponding group of fully connected layers to prune the output channel of each fully connected layer in the corresponding group of fully connected layers. Using the original transformation model and training samples, the transformation model to be trained is trained to obtain the pruning transformation model corresponding to the original transformation model; The original transformation model and the transformation model to be trained are used to process multimedia files; The training samples are multimedia files; multimedia files include one or more of images, videos, and audio.

2. The method according to claim 1, characterized in that, The encoder of the original transform model consists of a sequentially connected attention network and a feedforward network. The attention network includes a query fully connected layer, a key fully connected layer, a value fully connected layer, and a projection fully connected layer. The feedforward network includes a first fully connected layer and a second fully connected layer connected to the first fully connected layer. It is required that the query fully connected layer, the key fully connected layer, and the value fully connected layer have the same number of output channels, and that the projection fully connected layer and the second fully connected layer have the same number of output channels. In the encoder of the original transform model, fully connected layers with related relationships are grouped into a set of fully connected layers, including: The query fully connected layer, the key fully connected layer, and the value fully connected layer are divided into a group of fully connected layers; The first fully connected layer is divided into a separate group of fully connected layers; The projected fully connected layer and the second fully connected layer are divided into a group of fully connected layers.

3. The method according to claim 1, characterized in that, The step of training the transformation model to be trained using the original transformation model and training samples includes: The training samples are processed using the original transformation model and the transformation model to be trained, respectively, to obtain a first processing result corresponding to the original transformation model and a second processing result corresponding to the transformation model to be trained. Based on the difference loss function between the first processing result and the second processing result, the mask layer corresponding to each fully connected layer in the encoder of the transformation model to be trained is updated.

4. The method according to claim 3, characterized in that, The difference loss function includes at least one or more of the following: mean squared error loss function, relative entropy loss function, and adversarial loss function.

5. The method according to claim 3, characterized in that, The difference loss function includes a relative entropy loss function. Both the first processing result and the second processing result are probability distributions corresponding to the training samples. The relative entropy loss function is: Among them, L KL Let P be the relative entropy loss function, n be the number of probability values ​​contained in the first processing result and the second processing result respectively, and P be the relative entropy loss function. i Let Q be the i-th probability value in the first processing result. i P is the i-th probability value in the second processing result. i With Q i The probability value belonging to the same dimension.

6. The method according to claim 3, characterized in that, The difference loss function includes an adversarial loss function. Before updating the mask layer corresponding to each fully connected layer in the encoder of the transform model to be trained based on the difference loss function between the first processing result and the second processing result, the method further includes: The first processing result and the second processing result are input into the decision-maker for reliability determination, and the adversarial loss function is generated based on the decision result.

7. The method according to claim 3, characterized in that, The difference loss function includes the mean squared error loss function, the relative entropy loss function, and the adversarial loss function. Updating the mask layer corresponding to each fully connected layer in the encoder of the transform model to be trained, based on the difference loss function between the first processing result and the second processing result, includes: The sum of the mean squared error loss function, the relative entropy loss function, and the adversarial loss function is calculated to obtain the overall loss function; Based on the overall loss function, update the mask layer corresponding to each fully connected layer in the encoder of the transformation model to be trained.

8. A model pruning device, characterized in that, include: The generation module is used to divide the fully connected layers with correlations into groups of fully connected layers in the encoder of the original transform model, and to pre-set a corresponding mask layer for each group of fully connected layers to generate the transform model to be trained. Each mask layer acts on each fully connected layer within its corresponding group to prune the output channels of each fully connected layer within that group. The training module is used to train the transformation model to be trained using the original transformation model and training samples to obtain the pruning transformation model corresponding to the original transformation model. The original transformation model and the transformation model to be trained are used to process multimedia files; The training samples are multimedia files; multimedia files include one or more of images, videos, and audio.

9. A model pruning device, characterized in that, include: Processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute one or more computer programs stored in the memory to implement the model pruning method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the model pruning method as described in any one of claims 1-7.

Images