A hybrid dynamic pruning method for large medical model compression

Abstract

The present invention discloses a hybrid dynamic pruning method for large medical model compression, which adopts a sparse predictor based on hybrid sparse prediction, and has the advantage of reducing performance loss compared with other dynamic pruning methods based solely on dynamic sparse prediction. It effectively solves the problem of serious performance degradation after compression of large medical models, improves the model reasoning speed, and provides a feasible solution for the promotion and application of high-sparse, high-performance large models in practical applications.

Description

Technical Field

[0001] The present invention belongs to the technical field of model compression, and in particular relates to a hybrid dynamic pruning method for large medical model compression. Background Art

[0002] Medical big models, as an advanced artificial intelligence technology, are becoming an important auxiliary tool for doctors in their work. Based on massive medical data and deep learning technology, it can provide precise support in a variety of medical application scenarios. It can help clinicians quickly retrieve relevant medical literature and diagnosis and treatment guidelines, thereby accelerating the decision-making process. Medical big models can analyze millions of scientific research papers and medical guidelines, and provide the latest research progress and treatment recommendations that match the patient's condition in just a few seconds. This instant information greatly improves the accuracy of medical diagnosis and treatment efficiency, especially when dealing with complex and rare diseases.

[0003] In addition, medical big models can understand and respond to patients' natural language queries, providing a range of services including health consultation, symptom assessment, appointment reminders, and medication guidance. Virtual health assistants further help doctors track patients' health status and treatment responses by collecting feedback from patients, thereby optimizing treatment plans. These applications not only improve the quality and efficiency of medical services, but also enhance the interaction between doctors and patients, providing patients with more personalized and comprehensive health management. However, in resource-constrained deployment situations, their huge size and computing requirements pose great challenges to actual deployment. As these challenges become more targeted, the field of model compression has become a key research area to alleviate these limitations.

[0004] Model pruning is the most common and effective method in the field of model compression. Traditional model pruning methods are mostly applied to ordinary neural networks (such as convolutional neural networks, recurrent neural networks, long short-term memory networks, etc.), evaluating the importance of convolution kernels and weights and then setting thresholds, and physically or logically removing convolution kernels and weights below the threshold. However, with the development of deep learning, the emergence of transformer network structures has brought new challenges to model pruning. In a general sense, large models are stacked based on the encoder or decoder components of transformers. These components themselves do not have so-called convolution kernels, but another new model mechanism, namely the self-attention mechanism. After large models attracted the attention of many researchers, the field of large model pruning came into being. Large model pruning can prune the attention heads in the self-attention mechanism or the channels in the feedforward neural network layer.

[0005] Although pruning large models can reduce hardware requirements when deploying models, and pruning has been well studied, pruning methods have not been widely used in large medical models due to high requirements for model performance. First, since general pruning methods have a large impact on model performance, and the general remedy is to retrain the pruned model, it is expensive to retrain the compressed model at a parameter scale of hundreds of billions. Second, large medical models have high requirements for model performance, and excessive performance degradation is unacceptable. Finally, compressed large medical models need to provide rapid inference acceleration feedback without relying on specific hardware. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the hybrid dynamic pruning method for large medical model compression provided by the present invention solves the problem that traditional pruning methods are difficult to apply to the large medical model compression process with high requirements on model performance.

[0007] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a hybrid dynamic pruning method for large medical model compression, comprising the following steps:

[0008] S1. Construct a large medical model; each large model block in the large medical model includes a multi-head attention layer and a feedforward neural network layer;

[0009] S2. After embedding and position encoding the input text of the medical large model, the text is used as the first training data of the medical large model for forward reasoning, and the attention head sequence and the neuron sequence are saved during the forward reasoning process; wherein the attention head sequence, the neuron sequence and the input text constitute the second training data;

[0010] S3, constructing a hybrid sparse predictor, including a dynamic sparse predictor and a static sparse predictor;

[0011] S4. Train the dynamic sparse predictor using the second training data, and predict and output the dynamic importance rankings of all attention heads in each multi-head attention layer and all neurons in the feedforward neural network layer;

[0012] S5, using a static sparse predictor to predict and output the static importance ranking of all attention heads in each multi-head attention layer and all neurons in the feedforward neural network layer;

[0013] S6. According to the dynamic importance sorting and the static importance sorting, the sparse masks of the large model blocks at each layer are determined, and the medical large model is pruned based on the sparse masks to obtain the compressed medical large model.

[0014] Furthermore, in step S2, the embedding coding formula is:

[0015] x=IE(I)

[0016] In the formula, represents the input text, and the superscript T represents the dimension of the input text. represents the output vector after embedding encoding, d model Represents the dimension of the embedded layer IE;

[0017] The formula for the position encoding is:

[0018] z=PE(x)

[0019] In the formula, The output feature after position encoding, i.e., the first training data, includes text information and position information.

[0020] Furthermore, in step S2, the method for obtaining the second training data is specifically:

[0021] In the forward reasoning process of the first training data x, according to the output h = MHA(x) of the multi-head attention layer, the L2 norm of each attention head is recorded. seq = L2(h), and based on it, save the required number of attention heads to form an attention head sequence;

[0022] For the output of the feedforward neural network layer out = FFN (h), record the L2 norm of each neuron in a certain dimension neural seq = L2(out), and form a neuron sequence according to the neurons that store the required data;

[0023] The attention head sequence, the neuron sequence, and the input text of the medical large model are combined into the second training data;

[0024] Among them, MHA(·) represents the multi-head attention layer, FFN(·) represents the feedforward neural network layer, and L2(·) represents the regularization calculation.

[0025] Further, in step S3, the dynamic sparse predictor includes two fully connected layers connected in sequence;

[0026] The static sparse predictor is constructed according to the weight matrix of each layer of the large model block in the medical large model.

[0027] Furthermore, the step S4 is specifically as follows:

[0028] Using the second training data, respectively training the dynamic sparse predictor for the multi-head attention layer and the feedforward neural network layer to obtain a corresponding dynamic sparse predictor;

[0029] Among them, the expression for training the multi-head attention layer is:

[0030] dynamic_head seq =Pre MHA (x)

[0031] The expression for training a feedforward neural network layer is:

[0032] dynamic_neural seq =Pre MHA (out)

[0033] Where dynamic_head seq Indicates the dynamic importance index of the attention head, dynamic_neural seq Represents the dynamic importance index of neurons, Pre MHA (·) represents the predictor function;

[0034] The trained dynamic sparse predictor is used to dynamically sort the attention heads and neurons in the corresponding multi-head attention layer and feedforward neural network layer.

[0035] Furthermore, in the medical big model, the reasoning mechanism of the multi-head attention layer is:

[0036]

[0037] The inference mechanism of the feedforward neural network layer is:

[0038] FNN(x)=W D ReLu(W U x)

[0039] Where Att (i) (x) represents the output of the i-th attention head, All represent matrices, Respectively represent W D is the weight matrix from the hidden layer to the output layer, the weight matrix from the input layer to the hidden layer, C m represents the mask matrix, d h W K , Softmax(·) represents the Softmax function, ReLU(·) represents the nonlinear activation function, and the superscript i represents the attention head index.

[0040] Furthermore, in step S5, the mechanism of the static sparse predictor is: the importance of the attention heads and neurons in each layer of the large model blocks is calculated by the L2 norm of the weight matrix of each layer of the large model blocks, and static importance ranking is performed on them;

[0041] Among them, for the multi-head attention layer in the large model block, use WV The importance of the attention head is evaluated and its weight matrix is ​​reorganized into Then calculate the L2 norm of the corresponding attention head to get the static importance index of the attention head static_head seq , whose expression is:

[0042] static_head seq =L2(Reshape(W V ))

[0043] For the feed-forward neural network layers in the large model block, W D The importance of each neuron is evaluated according to the L2 norm, and the static importance index of the neuron is obtained, which is expressed as:

[0044] static_neural seq =L2(W D )

[0045] Among them, H represents the number of attention heads and d represents the attention dimension.

[0046] Furthermore, the step S6 is specifically as follows:

[0047] S61, mixing the dynamic importance ranking and the static importance ranking, and removing the attention heads and neurons corresponding to the importance indicators predicted by the dynamic sparse predictor and the static sparse predictor that are lower than the preset threshold;

[0048] S62, according to the set sparsity of the large model blocks of each layer of the medical large model, sort the importance of the remaining attention heads and neurons to generate sparse masks of the large model blocks of each layer;

[0049] S63. Prune the attention heads and neurons in the large medical model according to the generated sparse mask to obtain a compressed large medical model.

[0050] Furthermore, in step S62, the formula for generating the sparse mask is:

[0051] M head =(topk-len(M remove ))(λstatic_head seq +(1-λ)dynamic_head seq )

[0052] M neural =(topk-len(M remove ))(λstatic_neural seq +(1

[0053] -λ)dynamic_neural seq )

[0054] Where M head represents the sparse mask of the attention head, M neural represents the sparse mask of the neuron, λ represents the set hyperparameter, topk is the set pruning threshold, M remove Indicates the location of the weight parameter actually removed, which is expressed as:

[0055] M remove ={seq|seq∈static seq And seq∈dynamic seq}

[0056] Among them, static seq and dynamic seq Represents static importance sequences and dynamic importance sequences, including attention heads and neurons.

[0057] Furthermore, in step S63, the multi-head attention layer in the pruned medical model is expressed as:

[0058] MHA(x)=M head Att(x)

[0059] The feedforward neural network layer in the pruned medical large model is represented as:

[0060] FFN(x)=WDM neural ReLU(WUx)

[0061] Where Att(x) represents the attention head, WU and WD are the feedforward up-projection layer and feedforward down-projection layer in the feedforward neural network layer, respectively.

[0062] The beneficial effects of the present invention are:

[0063] 1. Compared with the existing technology, the present invention targets the high-efficiency and high-speed performance requirements of large medical models. In the case of high sparsity, the performance of the model itself is less reduced, so that the final large medical model has the dual advantages of high sparsity and high performance. The idea of ​​structured pruning is used to further reduce the deployment difficulty and improve the model reasoning speed.

[0064] 2. The method of the present invention can effectively solve the problem of serious performance degradation of large medical models after compression, and improve the model reasoning speed, providing a feasible solution for the promotion and application of highly sparse and high-performance large models in practical applications.

[0065] 3. The sparse predictor based on hybrid sparse prediction is adopted in this invention, which has the advantage of reducing performance loss compared with other dynamic pruning methods based solely on dynamic sparse prediction. In addition, thanks to the structural understanding of large models, this method has a wide range of applicability. This technology adopts the method of lightweight model fine-tuning components and can be applied to various types of models and tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flow chart of a hybrid dynamic pruning method for large medical model compression is provided for an embodiment of the present invention.

[0067] Figure 2 A schematic diagram of a hybrid sparse predictor provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0069] The embodiment of the present invention provides a hybrid dynamic pruning method for large medical model compression, such as Figure 1 As shown, the following steps are included:

[0070] S1. Construct a large medical model; each large model block in the large medical model includes a multi-head attention layer and a feedforward neural network layer;

[0071] S2. After embedding and position encoding the input text of the medical large model, the text is used as the first training data of the medical large model for forward reasoning, and the attention head sequence and the neuron sequence are saved during the forward reasoning process; wherein the attention head sequence, the neuron sequence and the input text constitute the second training data;

[0072] S3, constructing a hybrid sparse predictor, including a dynamic sparse predictor and a static sparse predictor;

[0073] S4. Train the dynamic sparse predictor using the second training data, and predict and output the dynamic importance rankings of all attention heads in each multi-head attention layer and all neurons in the feedforward neural network layer;

[0074] S5, using a static sparse predictor to predict and output the static importance ranking of all attention heads in each multi-head attention layer and all neurons in the feedforward neural network layer;

[0075] S6. According to the dynamic importance sorting and the static importance sorting, the sparse masks of the large model blocks at each layer are determined, and the medical large model is pruned based on the sparse masks to obtain the compressed medical large model.

[0076] The general working process of the hybrid sparse predictor provided in this embodiment is as follows: Figure 2 As shown in the figure, the output data of the previous layer is input into the hybrid predictor. The hybrid predictor is divided into two parts. One is a dynamic sparse predictor, which inputs the input data into a sparse predictor composed of two fully connected layers. This sparse predictor needs to be trained before it is applied to reasoning. The training data is composed of the input of this layer and the activation neuron sequence of this layer during the forward propagation process. After the dynamic sparse predictor obtains the output data of the previous layer, it can give the activation neuron sequence of this layer. The input of the static sparse predictor is also the output data of the previous layer. The difference is that the static sparse predictor calculates the data sequence in the output data that contributes significantly to the entire data segment and records its sequence position. Then, the final sparse mask is obtained by combining the results obtained by the dynamic sparse predictor in a certain proportion. The mask is then applied to the model structure of this layer to reduce the model calculation cost and time cost, and ultimately improve the reasoning speed.

[0077] Specifically, in step S2 of the embodiment of the present invention, the embedded coding formula is:

[0078] x=IE(I)

[0079] In the formula, represents the input text, and the superscript T represents the dimension of the input text. represents the output vector after embedding encoding, d m o del Indicates the dimension of the embedding layer IE, that is, how long a vector needs to be used to represent each word;

[0080] The formula for the position encoding is:

[0081] z=PE(x)

[0082] In the formula, The output feature after position encoding, i.e., the first training data, includes text information and position information.

[0083] In step S2 of the embodiment of the present invention, the method for obtaining the second training data is specifically:

[0084] In the forward reasoning process of the first training data x, according to the output h = MHA(x) of the multi-head attention layer, the L2 norm of each attention head is recorded. seq = L2(h), and based on it, save the required number of attention heads to form an attention head sequence;

[0085] For the output of the feedforward neural network layer out = FFN (h), record the L2 norm of each neuron in a certain dimension neural seq = L2(out), and form a neuron sequence according to the neurons that store the required data;

[0086] The attention head sequence, the neuron sequence, and the input text of the medical large model are combined into the second training data;

[0087] Among them, MHA(·) represents the multi-head attention layer, FFN(·) represents the feedforward neural network layer, and L2(·) represents the regularization calculation.

[0088] After the above steps, we get a series of h and head seq The corresponding data of out and neural seq The corresponding data is used as the second training data of the dynamic sparse predictor.

[0089] In step S3 of the embodiment of the present invention, the dynamic sparse predictor includes two fully connected layers connected in sequence; the static sparse predictor is constructed according to the weight matrix of each layer of the large model block in the medical large model.

[0090] Specifically, in this embodiment, the structures of the multi-head attention layer MHA and the feedforward neural network layer MLP are similar, but they are slightly different due to different output dimensions, and are two predictors.

[0091] Based on this, step S4 of the embodiment of the present invention is specifically as follows:

[0092] Using the second training data, respectively training the dynamic sparse predictor for the multi-head attention layer and the feedforward neural network layer to obtain a corresponding dynamic sparse predictor;

[0093] Among them, the expression for training the multi-head attention layer is:

[0094] dynamic_head seg =Pre MHA (x)

[0095] The expression for training the feedforward neural network layer is:

[0096] dynamic_neural seg =Pre MHA (out)

[0097] Where dynamic_head seg Indicates the dynamic importance index of the attention head, dynamic_neural segRepresents the dynamic importance index of neurons, Pre MHA (·) represents the predictor function;

[0098] The trained dynamic sparse predictor is used to dynamically sort the attention heads and neurons in the corresponding multi-head attention layer and feedforward neural network layer.

[0099] For the static sparse predictor, because the mechanisms of MHA and FFN are different, they need to be processed separately. The reasoning mechanism of the multi-head attention layer in this embodiment is:

[0100]

[0101] The inference mechanism of the feedforward neural network layer is:

[0102] FNN(x)=W D ReLu(W U x)

[0103] All represent matrices, Respectively represent W D is the weight matrix from the hidden layer to the output layer, the weight matrix from the input layer to the hidden layer, C m represents the mask matrix, d h W K , Softmax(·) represents the Softmax function, ReLU(·) represents the nonlinear activation function, and the superscript i represents the attention head index.

[0104] Among them, d h It is W K The dimension of C allows the model to learn information from different representation subspaces. m Represents a mask matrix, which is used to control the attention allocation in the attention mechanism. The role of the mask matrix is ​​to mask some positions when calculating the attention weights so that the model can ignore certain positions or limit the scope of attention according to the requirements of the task when processing the sequence. The role of the Softmax function is to convert the attention score (attention weight) into an attention distribution so that the sum of the attention weights is 1 and the attention weight of each position is between 0 and 1. This ensures that the importance of each position is reasonably taken into account when the model performs weighted summation. W U x is the intermediate representation of the input feature x obtained by linear transformation. ReLU stands for Rectified Linear Unit, which is a nonlinear activation function used to introduce nonlinearity into the network. UThe negative values ​​in x are set to zero, allowing the network to learn more complex functions. FNN(x) is the final output obtained by another linear transformation of the intermediate representation after the ReLU activation function.

[0105] In step S5 of the embodiment of the present invention, the mechanism of the static sparse predictor is: the importance of the attention heads and neurons in each layer of the large model blocks is calculated by the L2 norm of the weight matrix of each layer of the large model blocks, and the static importance is sorted;

[0106] Among them, for the multi-head attention layer in the large model block, use W V The importance of the attention head is evaluated and its weight matrix is ​​reorganized into Then calculate the L2 norm of the corresponding attention head to get the static importance index of the attention head static_head seq , whose expression is:

[0107] static_head seq =L2(Reshape(W V ))

[0108] For the feed-forward neural network layers in the large model block, W D The importance of each neuron is evaluated according to the L2 norm, and the static importance index of the neuron is obtained, which is expressed as:

[0109] static_neural seq =L2(W D )

[0110] Among them, H represents the number of attention heads and d represents the attention dimension.

[0111] Step S6 of the embodiment of the present invention is specifically as follows:

[0112] S61, mixing the dynamic importance ranking and the static importance ranking, and removing the attention heads and neurons corresponding to the importance indicators predicted by the dynamic sparse predictor and the static sparse predictor that are lower than the preset threshold;

[0113] S62, according to the set sparsity of the large model blocks of each layer of the medical large model, sort the importance of the remaining attention heads and neurons to generate sparse masks of the large model blocks of each layer;

[0114] S63. Prune the attention heads and neurons in the large medical model according to the generated sparse mask to obtain a compressed large medical model.

[0115] In step S62 of this embodiment, the formula for generating the sparse mask is:

[0116] M head=(topk-len(M remove ))(λstatic_head seq +(1-λ)dynamic_head seq )

[0117] M neural =(topk-len(M remove ))(λstatic_neural seq +(1

[0118] -λ)dynamic_neural seg )

[0119] Where M head represents the sparse mask of the attention head, M neural represents the sparse mask of neurons, λ represents the hyperparameters set, and M remove Indicates the location of the weight parameter actually removed, which is expressed as:

[0120] M remove ={seq|seq∈static seq And seq∈dynamic seq}

[0121] Among them, static seq and dynamic seq Represents static importance sequences and dynamic importance sequences, including attention heads and neurons.

[0122] In this embodiment, the final mask is obtained through importance sorting and threshold screening.

[0123] In step S63 of this embodiment, the multi-head attention layer in the pruned medical large model is expressed as:

[0124] MHA(x)=M head Att(x)

[0125] The feedforward neural network layer in the pruned medical large model is represented as:

[0126] FFN(x)=WDM neural ReLU(WUx)

[0127] Where Att(x) represents the attention head, WU and WD are the feedforward up-projection layer and feedforward down-projection layer in the feedforward neural network layer, respectively.

[0128] In this embodiment, a lightweight, easy-to-deploy, and efficient medical model is obtained through the above-mentioned model compression steps.

[0129] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0130] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A hybrid dynamic pruning method for large medical model compression, characterized in that: The following steps are involved: S1. Construct a large medical model; each large model block in the large medical model includes a multi-head attention layer and a feedforward neural network layer; S2. After embedding and position encoding the input text of the medical large model, the text is used as the first training data of the medical large model for forward reasoning, and the attention head sequence and the neuron sequence are saved during the forward reasoning process; wherein the attention head sequence, the neuron sequence and the input text constitute the second training data; S3, constructing a hybrid sparse predictor, including a dynamic sparse predictor and a static sparse predictor; S4. Train the dynamic sparse predictor using the second training data, and predict and output the dynamic importance rankings of all attention heads in each multi-head attention layer and all neurons in the feedforward neural network layer; S5, using a static sparse predictor to predict and output the static importance ranking of all attention heads in each multi-head attention layer and all neurons in the feedforward neural network layer; S6. Determine the sparse masks of the large model blocks at each layer according to the dynamic importance ranking and the static importance ranking, and prune the medical large model according to the sparse masks to obtain a compressed medical large model; The step S4 is specifically as follows: Using the second training data, respectively training the dynamic sparse predictor for the multi-head attention layer and the feedforward neural network layer to obtain a corresponding dynamic sparse predictor; Among them, the expression for training the multi-head attention layer is: The expression for training the feedforward neural network layer is: In the formula, represents the dynamic importance index of the attention head, represents the dynamic importance index of the neuron, represents the predictor function; The trained dynamic sparse predictor is used to dynamically sort the attention heads and neurons in the corresponding multi-head attention layer and feedforward neural network layer; In step S5, the mechanism of the static sparse predictor is: the importance of the attention heads and neurons in each layer of the large model blocks is calculated by the L2 norm of the weight matrix of each layer of the large model blocks, and static importance ranking is performed on them; Among them, for the multi-head attention layer in the large model block, use The importance of the attention head is evaluated and its weight matrix is ​​reorganized into , and then calculate the L2 norm of the corresponding attention head to get the static importance index of the attention head , whose expression is: For the feed-forward neural network layers in the large model block, The importance of each neuron is evaluated according to the L2 norm, and the static importance index of the neuron is obtained, which is expressed as: in, H represents the number of attention heads, d Represents the attention dimension.

2. The hybrid dynamic pruning method for large medical model compression according to claim 1, characterized in that: In step S2, the embedding coding formula is: In the formula, represents the input text, and the superscript T represents the dimension of the input text. represents the output vector after embedding encoding, Representation Embedding Layer Dimensions; The formula for the position encoding is: In the formula, The output feature after position encoding, i.e., the first training data, includes text information and position information.

3. The hybrid dynamic pruning method for large medical model compression according to claim 1, characterized in that: In step S2, the method for obtaining the second training data is specifically: In the first training data In the forward reasoning process, according to the output of the multi-head attention layer , record the L2 norm of each attention head , and form an attention head sequence based on the required number of attention heads saved; For the output of the feedforward neural network layer , record the L2 norm of each neuron in a certain dimension , and form a neuron sequence according to the neurons that store the required data; The attention head sequence, the neuron sequence, and the input text of the medical large model are combined into the second training data; in, represents the multi-head attention layer, represents a feed-forward neural network layer, Represents the regularization calculation.

4. The hybrid dynamic pruning method for large medical model compression according to claim 3 is characterized in that: In step S3, the dynamic sparse predictor includes two fully connected layers connected in sequence; The static sparse predictor is constructed according to the weight matrix of each layer of the large model block in the medical large model.

5. The hybrid dynamic pruning method for large medical model compression according to claim 1, characterized in that: In the medical big model, the reasoning mechanism of the multi-head attention layer is: The inference mechanism of the feedforward neural network layer is: In the formula, Indicates i The output of an attention head is All represent matrices, Respectively is the weight matrix from the hidden layer to the output layer, and the weight matrix from the input layer to the hidden layer, represents the mask matrix, express The dimension of represents the Softmax function, represents a nonlinear activation function, the superscript i Represents the attention head index.

6. The hybrid dynamic pruning method for large medical model compression according to claim 1, characterized in that: The step S6 is specifically as follows: S61, mixing the dynamic importance ranking and the static importance ranking, and removing the attention heads and neurons corresponding to the importance indicators predicted by the dynamic sparse predictor and the static sparse predictor that are lower than the preset threshold; S62, according to the set sparsity of the large model blocks of each layer of the medical large model, sort the importance of the remaining attention heads and neurons to generate sparse masks of the large model blocks of each layer; S63. Prune the attention heads and neurons in the large medical model according to the generated sparse mask to obtain a compressed large medical model.

7. The hybrid dynamic pruning method for large medical model compression according to claim 6, characterized in that: In step S62, the formula for generating the sparse mask is: In the formula, represents the sparse mask of the attention head, represents the sparse mask of neurons, represents the hyperparameters set, topk is the set pruning threshold, Indicates the location of the weight parameter actually removed, which is expressed as: in, and Represents static importance sequences and dynamic importance sequences, including attention heads and neurons.

8. The hybrid dynamic pruning method for large medical model compression according to claim 7, characterized in that: In step S63, the multi-head attention layer in the pruned medical model is expressed as: The feedforward neural network layer in the pruned medical large model is represented as: In the formula, represents the attention head, and They are respectively the feedforward up-projection layer and the feedforward down-projection layer in the feedforward neural network layer.

Images