Small sample learning-based medical examination tutoring large model fine adjustment method and system

Abstract

The invention discloses a medical examination tutoring large model fine tuning method and system based on small sample learning, and the method comprises the steps: employing clustering and attention scores of word embedding vectors, selecting five word embedding vectors as a data set to train a medical examination tutoring large model, and storing the weight of an adapter in the training process; performing pruning operation on the weight of the adapter according to the importance index of the weight and a pruning threshold value, and performing reasoning test on the medical examination tutoring large model before and after pruning to obtain reasoning time and test scores before and after pruning; according to the reasoning time and the test score, calculating an accuracy reduction index and an acceleration performance index of the medical examination tutoring large model before and after pruning; and judging whether the accuracy rate reduction index and the acceleration performance index or the pruning times meet corresponding preset conditions, if so, obtaining a fine-adjusted medical examination tutoring large model, otherwise, updating a pruning threshold according to the accuracy rate reduction index and the acceleration performance index, and then returning to the pruning operation step.

Description

Technical Field

[0001] The present invention relates to the technical field of question-answering models, and in particular to a method and system for fine-tuning a large model for tutoring medical examinations based on small sample learning. Background Art

[0002] The Dental Practitioner Examination, as an important assessment in the medical field, aims to comprehensively and objectively evaluate whether applicants for dentist qualifications truly possess the professional knowledge and skills necessary to engage in dentistry. The Dental Practitioner Examination covers many aspects of dentistry, including the diagnosis, treatment, and prevention of oral diseases, as well as knowledge in multiple professional fields such as oral and maxillofacial surgery, prosthodontics, and orthodontics. Candidates need to systematically master this professional knowledge and be able to skillfully apply it to clinical practice to provide patients with high-quality dental medical services.

[0003] In the written examination of the dental practitioner examination, there are 600 multiple-choice questions that candidates need to answer within two and a half hours. These questions cover the following three aspects: (1) Basic theoretical knowledge of stomatology: including basic knowledge such as oral anatomy, oral physiology, and oral pathology. (2) Oral clinical knowledge: including diagnosis and treatment knowledge of clinical subjects such as oral medicine, oral and maxillofacial surgery, oral rehabilitation, and oral orthodontics. (3) Comprehensive medical knowledge: including medical microbiology, medical immunology, pharmacology, medical psychology, medical ethics, health regulations, etc. These exams cover many subjects, a wide range of knowledge points, and a short examination time, so it is extremely difficult to pass the exam.

[0004] In order to pass the written test, most candidates improve their scores by attending offline tutoring classes or doing a lot of exercises. However, this method has the following disadvantages. First, attending offline tutoring classes or doing a lot of exercises requires candidates to invest a lot of time and energy. This high-intensity learning method can easily make candidates feel tired, resulting in low learning efficiency. Secondly, offline tutoring classes and exercise books are often aimed at a large number of candidates, and the content design is relatively broad, making it difficult to provide personalized teaching based on the specific situation of each candidate. Therefore, candidates will inevitably spend a lot of time on the content they have mastered, while ignoring their weak links that they have not mastered.

[0005] In recent years, large language models have shown great potential in various fields, especially in the field of education, where models that can support personalized teaching have begun to emerge. Specifically, through the existing large model training technology, it is easy to use a large amount of oral professional books and materials as input data to build a large prediction model with oral knowledge. However, this model cannot answer the content of the test questions well. For example: if the model is asked to choose from four options, the model will give a wrong answer, but when it is asked to further analyze the test questions, it can give a correct analysis. That is, although the existing model has oral knowledge, it cannot apply this knowledge well to achieve tutoring. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the method and system for fine-tuning a large model for medical examination tutoring based on small sample learning provided by the present invention solves the problem of low accuracy in answering questions by the existing large prediction model with oral knowledge.

[0007] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0008] In a first aspect, a method for fine-tuning a large model for medical examination tutoring based on small sample learning is provided, which comprises the steps of:

[0009] S1. Obtain oral knowledge test questions and convert the test questions into word embedding vectors using word embedding;

[0010] S2. Cluster the word embedding vectors, calculate the attention score between each word embedding vector and the cluster center, and select the word embedding vectors whose attention scores meet the preset conditions as the data set;

[0011] S3, training a medical examination coaching model using the data set and storing the weights of the adapter during the training process, wherein the medical examination coaching model is composed of a Transformer layer embedded with the adapter;

[0012] S4. Prune the weights of the adapter according to the importance index of the weight and the pruning threshold, and then perform reasoning tests on the medical examination tutoring model before and after pruning to obtain the reasoning time and test scores before and after pruning;

[0013] S5. Calculate the accuracy drop index and acceleration performance index of the medical examination tutoring model before and after pruning based on the inference time and test score before and after pruning;

[0014] S6, judging whether the accuracy drop index and the acceleration performance index or the number of pruning times meet the corresponding preset conditions, if so, obtaining the fine-tuned medical examination tutoring model, otherwise proceeding to step S7;

[0015] S7. Update the pruning threshold according to the accuracy drop index and the acceleration performance index, and then return to step S4.

[0016] Furthermore, the expression for pruning the adapter weights is:

[0017]

[0018]

[0019] Among them, W s is the weight matrix after pruning; M is a sparse matrix; W is the weight matrix before pruning; m and n are the total number of rows and columns of the adapter's weight matrix, respectively; W de is the weight parameter of the dth row and eth column in the adapter’s weight matrix; w de_matic W de The importance of the index; init is the pruning threshold; W d is the weight parameter of the dth row in the adapter’s weight matrix; W de_f is the value of the weight parameter at the fth word embedding vector; F is the total number of word embedding vectors in the dataset.

[0020] Furthermore, the expressions for calculating the accuracy drop index and the acceleration performance index are:

[0021]

[0022]

[0023] in, It is an indicator of accuracy decline; and These are the test scores of the large model for medical examination tutoring before and after pruning; and They are the reasoning time of the large model for medical examination tutoring before and after pruning; To accelerate performance indicators.

[0024] Furthermore, the expression for updating the pruning threshold is:

[0025]

[0026] Among them, thershold update and Thershold init are the pruning thresholds before and after updating respectively; a and b are weight coefficients.

[0027] Furthermore, the Transformer layer includes an embedding layer, a multi-head attention, a feedforward layer and an adapter connected in sequence, and a layer normalization, two feedforward layers, an adapter and a normalization layer connected in sequence, and the output of the embedding layer and the output of the adapter adjacent to it are superimposed and the input layer is normalized; a dropout layer is provided between the adapter and the hidden layer;

[0028] The adapter includes an input layer and an output layer, wherein a plurality of hidden layers are arranged between the input layer and the output layer, and the hidden layers map the input features to a high-dimensional representation space through a plurality of linear transformations and nonlinear activation functions, and generate a final output.

[0029] Furthermore, the expression for multi-head attention processing the output of the embedding layer is:

[0030]

[0031] Where z is the output of the embedding layer; W Q , W K , W V , W O are all Transformer layer weight matrices; d h W K Dimension; G m is the mask matrix; Softmax(.) is the Softmax function; q is the qth head of the multi-head attention; T is the transpose;

[0032] The expression for the feedforward layer to process the input feature vector is:

[0033] FNN(o)=W D ReLu(W U o)

[0034] Where o is the feature vector of the input feedforward layer; W U o is the intermediate representation of o obtained by linear transformation; W U is the weight matrix from the input layer to the hidden layer; ReLU is the rectified linear unit; FNN(o) is the final output obtained by linear transformation of the intermediate representation after the ReLU activation function; W D is the weight matrix from the hidden layer to the output layer;

[0035] The expression for layer normalization to process the input feature vector is:

[0036]

[0037] Where LayerNorm(p) is the output of layer normalization; p is the normalized feature vector of the input layer; μ and σ are the mean and standard deviation of p, respectively; γ and β are the learnable scaling factor and bias term, respectively;

[0038] The expression used by the adapter to process the input feature vector is:

[0039] Adapter(t)=Dropout(W2·(ReLU(W1·t)))

[0040] Where Adapter(t) is the output of the adapter; t is the feature vector of the input adapter; W1 and W2 are the weight matrices of the two linear transformations of the adapter respectively; Dropout is the dropout operation.

[0041] Furthermore, before using the dataset to train the large model for medical examination tutoring, the weight information of the non-adapter part in the Transformer layer needs to be frozen, and only the weight of the adapter is updated during the training process.

[0042] Furthermore, the method of clustering word embedding vectors includes:

[0043] A1. Randomly select K cluster centers from all word embedding vectors and perform K-means clustering:

[0044]

[0045] Among them, μ j The jth cluster center selected by K-means; c i is the word embedding vector x i Index into the cluster; ||x i -μ j || 2 is the word embedding vector x i to μ j The square of the Euclidean distance;

[0046] A2. Calculate the mean of all word embedding vectors in each cluster as the new cluster center:

[0047]

[0048] Among them, \C j \ is cluster C j The number of data points; is the new cluster center;

[0049] A3: Determine whether the change between the current cluster center of each cluster and the cluster center of the last clustering is less than the preset threshold. If so, complete the clustering. Otherwise, update And return to step A1.

[0050] The method of calculating the attention score of each word embedding vector and cluster center includes:

[0051] B1. Calculate the cosine similarity between each word embedding vector and the cluster center:

[0052]

[0053] Among them, S ji is the word embedding vector x i and cluster C j The cluster center μ j The cosine similarity of ; ||·|| is the Euclidean norm of the vector;

[0054] B2. Calculate the attention score of each word embedding vector based on cosine similarity:

[0055]

[0056] Among them, A ij For x i Relative to the cluster center μ j Attention score; S ki For x i and cluster center μ k The cosine similarity of , k is a variable, 1≤k≤K, and exp(.) is a natural exponential function.

[0057] Furthermore, the method of selecting a word embedding vector whose attention score meets a preset condition as a data set includes:

[0058] C1. Sort the word embedding vectors in each cluster in descending order according to their attention scores, and select the word embedding vector with the highest attention score in each cluster;

[0059] C2. Sort all selected word embedding vectors in descending order according to their attention scores, and select a preset number of word embedding vectors with the highest attention scores to form a data set.

[0060] In a second aspect, a large model fine-tuning system for medical examination tutoring based on small sample learning is provided, which includes a word embedding module, an oral test question small sample extraction component and a lightweight model fine-tuning component connected in sequence, and the lightweight model fine-tuning component includes a large model training module, a pruning and reasoning test module, an indicator calculation module, a judgment module and a threshold update module connected in sequence;

[0061] The word embedding module is used to obtain oral knowledge test questions and convert the test questions into word embedding vectors using word embedding;

[0062] The oral test small sample extraction component is used to cluster word embedding vectors and calculate the attention score between each word embedding vector and the cluster center, and select the word embedding vectors whose attention scores meet the preset conditions as the data set;

[0063] The large model training module is used to train the medical examination tutoring large model using the data set and store the weights of the adapter during the training process, wherein the medical examination tutoring large model is composed of a Transformer layer embedded with an adapter;

[0064] The pruning and reasoning test module is used to prune the weights of the adapter according to the importance index of the weights and the pruning threshold, and then perform reasoning tests on the medical examination tutoring model before and after pruning to obtain the reasoning time and test scores before and after pruning;

[0065] The indicator calculation module is used to calculate the accuracy drop index and acceleration performance index of the medical examination tutoring model before and after pruning based on the inference time and test score before and after pruning;

[0066] The judgment module is used to judge whether the accuracy drop index and the acceleration performance index or the number of pruning times meet the corresponding preset conditions. If so, the fine-tuned medical examination tutoring model is obtained, otherwise, the threshold update module is entered;

[0067] The threshold update module is used to update the pruning threshold according to the accuracy drop index and the acceleration performance index, and then return to the pruning and reasoning test module.

[0068] The beneficial effects of the present invention are as follows: when extracting small samples, this solution not only finds the most representative sample data from the massive data based on the application of clustering and attention scores, but also uses the attention mechanism to focus on the key information in each type of sample, and uses these key information to better select representative samples, greatly reducing the annotation cost of sample extraction. When extracting samples, a small number of representative oral test questions are extracted for subsequent fine-tuning, which not only greatly reduces the cost of manual screening of test questions, but also improves the generalization ability and training efficiency of the model for different data.

[0069] When fine-tuning the model, this solution reduces the number of model parameters through lightweight operations (pruning operations), improves the efficiency of data and models, and has wide applicability, making the model lighter and easier to deploy and apply. In addition, through multiple cycles of iterations in the pruning process, the pruning threshold is continuously updated, and an excellent model that takes into account both accuracy and inference speed can be found. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 Flowchart of the method for fine-tuning the large model for medical examination tutoring based on small sample learning.

[0071] Figure 2 Schematic of fine-tuning components for lightweight models.

[0072] Figure 3 This is the principle block diagram of the large model fine-tuning system for medical examination tutoring based on small sample learning. DETAILED DESCRIPTION

[0073] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0074] refer to Figure 1 , Figure 1 A method for fine-tuning a large model for medical examination tutoring based on small sample learning is shown; Figure 1 As shown, the method S includes steps S1 to S7.

[0075] In step S1, oral knowledge test questions are obtained, and the test questions are converted into word embedding vectors using word embedding;

[0076] In step S2, the word embedding vectors are clustered, and the attention score between each word embedding vector and the cluster center is calculated, and the word embedding vectors whose attention scores meet the preset conditions are selected as the data set;

[0077] During implementation, the method for clustering word embedding vectors preferably includes:

[0078] A1. Randomly select K cluster centers from all word embedding vectors and perform K-means clustering:

[0079]

[0080] Among them, μ j The jth cluster center selected by K-means; c i is the word embedding vector x i Index into the cluster; ||x i -μ j || 2 is the word embedding vector x i to μ j The square of the Euclidean distance;

[0081] A2. Calculate the mean of all word embedding vectors in each cluster as the new cluster center:

[0082]

[0083] Among them, \C j \ is cluster C j The number of data points; is the new cluster center;

[0084] A3: Determine whether the change between the current cluster center of each cluster and the cluster center of the last clustering is less than the preset threshold. If so, complete the clustering. Otherwise, update And return to step A1.

[0085] This solution uses K-means to replace the randomly selected K cluster centers, making the cluster centers more representative and improving the convergence speed and effect. Each cluster center is updated iteratively and the question vectors are redistributed until the cluster center is stable or the preset number of iterations is reached. Finally, each question vector is assigned a cluster label to indicate the cluster to which it belongs.

[0086] The method of calculating the attention score of each word embedding vector and cluster center includes:

[0087] B1. Calculate the cosine similarity between each word embedding vector and the cluster center:

[0088]

[0089] Among them, S ji is the word embedding vector x i and cluster C j The cluster center μ j The cosine similarity of ; ||·|| is the Euclidean norm of the vector;

[0090] B2. Calculate the attention score of each word embedding vector based on cosine similarity:

[0091]

[0092] Among them, A ij For x i Relative to the cluster center μ j Attention score; S ki For x i and cluster center μ k The cosine similarity of , k is a variable, 1≤k≤K, and exp(.) is a natural exponential function.

[0093] In this scheme, cosine similarity is used to measure the directional relationship between two vectors rather than their absolute distance in space, which is very suitable for measuring the semantic relevance between word embeddings. Secondly, the similarity matrix calculated above is normalized by the softmax function to obtain the attention weight (score) of each word vector for each cluster center. This step ensures that the sum of all weights is 1, giving each word relative importance in different clustering dimensions. Finally, a weighted word representation can be generated by multiplying the word embedding vector of each word by its corresponding attention score. This weighted representation strengthens the word information in the text that is most relevant to the cluster center, reduces the impact of noise, and thus improves the performance of the model in subsequent tasks.

[0094] Methods for selecting word embedding vectors whose attention scores meet preset conditions as data sets include:

[0095] C1. Sort the word embedding vectors in each cluster in descending order according to their attention scores, and select the word embedding vector with the highest attention score in each cluster;

[0096] C2. Sort all selected word embedding vectors in descending order according to their attention scores, and select a preset number of word embedding vectors with the highest attention scores to form a data set. In this solution, the preferred preset number is 5.

[0097] The operation of step C1 ensures that representative questions in each cluster participate in the subsequent selection process. Step C2 aggregates the representative questions in all clusters and sorts them again. In this process, the focus is on the representativeness of the questions in the global scope to ensure that the small number of questions finally selected can fully cover the diversity and representativeness of the entire question bank.

[0098] When selecting data sets, this solution ensures that the most representative test questions are extracted efficiently and accurately from a large-scale test question bank through hierarchical sorting and screening, thereby optimizing the efficiency of data set construction and use.

[0099] In step S3, the medical examination tutoring model is trained using the data set, and the weights of the adapters during the training process are stored. The medical examination tutoring model consists of a Transformer layer embedded with an adapter. Before the medical examination tutoring model is trained using the data set, the weight information of the non-adapter part in the Transformer layer needs to be frozen, and only the weights of the adapters are updated during the training process.

[0100] like Figure 2As shown, during implementation, the preferred Transformer layer of this scheme includes an embedding layer, a multi-head attention, a feedforward layer and an adapter connected in sequence, and a layer normalization, two feedforward layers, an adapter and a normalization layer connected in sequence. The output of the embedding layer and the output of the adapter adjacent to it are superimposed and the input layer is normalized; a dropout layer is arranged between the adapter and the hidden layer.

[0101] The feedforward layer is a feedforward neural network (FFN) layer, which consists of two fully connected layers and a nonlinear change. In order to maintain the lower limit of model performance, a residual connection is also designed in the feedforward neural network (FFN) layer to ensure that the model performance will not degrade.

[0102] In one embodiment of the present invention, the multi-head attention mechanism first performs a linear transformation on the input sequence to obtain query, key, and value vectors. Then, the query, key, and value of each head are respectively input into the attention mechanism to calculate the attention weight of each head. Next, the attention weight of each head is multiplied by the corresponding value and weighted summed to obtain the output of each head. Finally, the outputs of all heads are spliced ​​together and subjected to another linear transformation to obtain the final output. The expression for the multi-head attention to implement the above process is:

[0103]

[0104] Where z is the output of the embedding layer; W Q , W K , W V , W O are all Transformer layer weight matrices; d h W K The dimension of G allows the model to learn information from different representation subspaces; m is the mask matrix; Softmax(.) is the Softmax function; q is the qth head of the multi-head attention; T is the transpose;

[0105] The mask matrix is ​​used to control the attention distribution in the attention mechanism. The role of the mask matrix is ​​to mask some positions when calculating the attention weights so that the model can ignore certain positions or limit the scope of attention according to the requirements of the task when processing the sequence. The role of the Softmax function is to convert the attention score (attention weight) into an attention distribution so that the sum of the attention weights is 1 and the attention weight of each position is between 0 and 1. This ensures that the importance of each position is reasonably taken into account when the model performs weighted summation.

[0106] The expression for the feedforward layer to process the input feature vector is:

[0107] FNN(o)=W D ReLu(W U o)

[0108] Where o is the feature vector of the input feedforward layer; W U o is the intermediate representation of o obtained by linear transformation; W U is the weight matrix from the input layer to the hidden layer; ReLU is the rectified linear unit, which is a nonlinear activation function used to introduce nonlinearity into the network; it inputs W U The negative parts of x are set to zero, so that the network can learn more complex functions. FNN(o) is the final output obtained by linear transformation of the intermediate representation after the ReLU activation function; W D is the weight matrix from the hidden layer to the output layer;

[0109] Layer Normalization is a normalization technique used in neural networks to accelerate the training process and improve the generalization ability of the model. The expression for processing the input feature vector is:

[0110]

[0111] Where LayerNorm(p) is the output of layer normalization; p is the normalized feature vector of the input layer; μ and σ are the mean and standard deviation of p, respectively; γ and β are the learnable scaling factor and bias term, respectively;

[0112] Reference again Figure 2 The adapter includes an input layer and an output layer, and several hidden layers are arranged between the input layer and the output layer. The hidden layers map the input features to a high-dimensional representation space through several linear transformations and nonlinear activation functions, and generate the final output.

[0113] The expression used by the adapter to process the input feature vector is:

[0114] Adapter(t)=Dropout(W2·(ReLU(W1·t)))

[0115] Where Adapter(t) is the output of the adapter; t is the feature vector of the input adapter; W1 and W2 are the weight matrices of the two linear transformations of the adapter respectively; Dropout is the dropout operation.

[0116] In this scheme, the adapter adds dropout connections between hidden layers compared to ordinary feedforward neural networks. The purpose of this is to introduce a certain degree of randomness during the fine-tuning process to prevent the model from overfitting and improve the generalization ability of the model.

[0117] In step S4, the weights of the adapter are pruned according to the importance index of the weights and the pruning threshold, and then the medical examination tutoring model before and after pruning is subjected to inference test to obtain the inference time and test score before and after pruning.

[0118] During implementation, the scheme preferably uses the expression for pruning the adapter weight as follows:

[0119]

[0120]

[0121] Among them, W s is the weight matrix after pruning, which has the same shape as W, and only the elements with corresponding positions of 1 in M ​​are retained. M is a sparse matrix with the same shape as W, in which the part greater than the initial threshold is 1 and the part less than the initial threshold is 0. W is the weight matrix before pruning; m and n are the total number of rows and columns of the adapter weight matrix respectively; W de is the weight parameter of the dth row and eth column in the adapter’s weight matrix; w de_matic W de The importance of the index; init is the pruning threshold; W d is the weight parameter of the dth row in the adapter’s weight matrix; W de_f is the value of the weight parameter at the fth word embedding vector; F is the total number of word embedding vectors in the dataset.

[0122] In step S5, based on the inference time and test score before and after pruning, the accuracy drop index and acceleration performance index of the medical examination tutoring model before and after pruning are calculated:

[0123]

[0124]

[0125] Among them, acc is the accuracy drop index; score dense and score spasity are the test scores of the medical examination tutoring model before and after pruning; t dense and t spasity They are the inference time of the large model for medical examination tutoring before and after pruning; speedup is the acceleration performance indicator.

[0126] In step S6, it is determined whether the accuracy drop index and the acceleration performance index or the number of pruning times meet the corresponding preset conditions. If so, a fine-tuned medical examination tutoring model is obtained, otherwise, step S7 is entered;

[0127] The specific operation of judging whether the accuracy drop index and the acceleration performance index or the number of pruning times meet the corresponding preset conditions is: when the accuracy drop index and the acceleration performance index are both less than their corresponding thresholds, the pruning operation is terminated, or when the number of pruning times is greater than or equal to the preset number, the pruning operation is terminated.

[0128] In step S7, the pruning threshold is updated according to the accuracy drop index and the acceleration performance index, and then the process returns to step S4. The expression for updating the pruning threshold is:

[0129]

[0130] Among them, thershold update and Thershold init are the pruning thresholds before and after updating respectively; a and b are weight coefficients.

[0131] like Figure 3 As shown, the present solution also provides a fine-tuning system for a large model for tutoring medical examinations based on small sample learning, which includes a word embedding module, an oral examination question small sample extraction component, and a lightweight model fine-tuning component connected in sequence, and the lightweight model fine-tuning component includes a large model training module, a pruning and reasoning test module, an indicator calculation module, a judgment module, and a threshold update module connected in sequence;

[0132] The word embedding module is used to obtain oral knowledge test questions and convert the test questions into word embedding vectors using word embedding;

[0133] The oral test small sample extraction component is used to cluster word embedding vectors and calculate the attention score between each word embedding vector and the cluster center, and select the word embedding vectors whose attention scores meet the preset conditions as the data set;

[0134] The large model training module is used to train the medical examination tutoring large model using the data set and store the weights of the adapter during the training process, wherein the medical examination tutoring large model is composed of a Transformer layer embedded with an adapter;

[0135] The pruning and reasoning test module is used to prune the weights of the adapter according to the importance index of the weights and the pruning threshold, and then perform reasoning tests on the medical examination tutoring model before and after pruning to obtain the reasoning time and test scores before and after pruning;

[0136] The indicator calculation module is used to calculate the accuracy drop index and acceleration performance index of the medical examination tutoring model before and after pruning based on the inference time and test score before and after pruning;

[0137] The judgment module is used to judge whether the accuracy drop index and the acceleration performance index or the number of pruning times meet the corresponding preset conditions. If so, the fine-tuned medical examination tutoring model is obtained, otherwise, the threshold update module is entered;

[0138] The threshold update module is used to update the pruning threshold according to the accuracy drop index and the acceleration performance index, and then return to the pruning and reasoning test module.

[0139] In summary, the fine-tuning method of this scheme can search for representative test questions as samples for fine-tuning, which not only reduces the amount of data required for fine-tuning, but also achieves multi-sample effects; when training, it combines pruning strategies to reduce the number of model parameters, making the model lighter and easier to deploy and apply.

Claims

1. A fine-tuning method for a large model for medical examination tutoring based on small sample learning, characterized in that: Includes steps: S1. Obtain oral knowledge test questions and convert the test questions into word embedding vectors using word embedding; S2. Cluster the word embedding vectors and calculate the attention score between each word embedding vector and the cluster center. Select the word embedding vectors whose attention scores meet the preset conditions as the data set. S3, training a medical examination coaching model using the data set and storing the weights of the adapter during the training process, wherein the medical examination coaching model is composed of a Transformer layer embedded with the adapter; S4. Prune the weights of the adapter according to the importance index of the weight and the pruning threshold, and then perform reasoning tests on the medical examination tutoring model before and after pruning to obtain the reasoning time and test scores before and after pruning; S5. Calculate the accuracy drop index and acceleration performance index of the medical examination tutoring model before and after pruning based on the inference time and test score before and after pruning; S6, judging whether the accuracy drop index and the acceleration performance index or the number of pruning times meet the corresponding preset conditions, if so, obtaining the fine-tuned medical examination tutoring model, otherwise proceeding to step S7; S7. Update the pruning threshold according to the accuracy drop index and the acceleration performance index, and then return to step S4.

2. The method for fine-tuning a large model for medical examination tutoring based on small sample learning according to claim 1 is characterized in that: The expression for pruning the adapter weights is: Among them, W s is the weight matrix after pruning; M is a sparse matrix; W is the weight matrix before pruning; m and n are the total number of rows and columns of the adapter's weight matrix, respectively; W de is the weight parameter of the dth row and eth column in the adapter’s weight matrix; w de_matic W de The importance of the index; init is the pruning threshold; W d is the weight parameter of the dth row in the adapter’s weight matrix; W de_f is the value of the weight parameter at the fth word embedding vector; F is the total number of word embedding vectors in the dataset.

3. The method for fine-tuning a large model for medical examination tutoring based on small sample learning according to claim 1 is characterized in that: The expressions for calculating the accuracy drop index and the acceleration performance index are: Among them, acc is the accuracy drop index; score dense and score spasity are the test scores of the medical examination tutoring model before and after pruning; t dense and t spasity They are the inference time of the large model for medical examination tutoring before and after pruning; speedup is the acceleration performance indicator.

4. The method for fine-tuning a large model for medical examination tutoring based on small sample learning according to claim 3 is characterized in that: The expression for updating the pruning threshold is: Among them, thershold update and Thershold init are the pruning thresholds before and after updating respectively; a and b are weight coefficients.

5. The method for fine-tuning a large model for tutoring medical examinations based on small sample learning according to any one of claims 1 to 4, characterized in that: The Transformer layer includes an embedding layer, a multi-head attention, a feedforward layer and an adapter connected in sequence, and a layer normalization, two feedforward layers, an adapter and a normalization layer connected in sequence. The output of the embedding layer and the output of the adapter adjacent to it are superimposed and the input layer is normalized; A dropout layer is provided between the adapter and the hidden layer; The adapter includes an input layer and an output layer, wherein a plurality of hidden layers are arranged between the input layer and the output layer, and the hidden layers map the input features to a high-dimensional representation space through a plurality of linear transformations and nonlinear activation functions, and generate a final output.

6. The method for fine-tuning a large model for medical examination tutoring based on small sample learning according to claim 5 is characterized in that: The expression for multi-head attention processing the output of the embedding layer is: Where z is the output of the embedding layer; W Q , W K , W V , W O are all Transformer layer weight matrices; d h W K Dimension; G m is the mask matrix; Softmax(.) is the Softmax function; q is the qth head of the multi-head attention; T is the transpose; The expression for the feedforward layer to process the input feature vector is: FNN(o)=W D ReLu (W U of) Where o is the feature vector of the input feedforward layer; W U o is the intermediate representation of o obtained by linear transformation; W U is the weight matrix from the input layer to the hidden layer; ReLU is the rectified linear unit; FNN(o) is the final output obtained by linear transformation of the intermediate representation after the ReLU activation function; W D is the weight matrix from the hidden layer to the output layer; The expression for layer normalization to process the input feature vector is: Where LayerNorm(p) is the output of layer normalization; p is the normalized feature vector of the input layer; μ and σ are the mean and standard deviation of p, respectively; γ and β are the learnable scaling factor and bias term, respectively; The expression used by the adapter to process the input feature vector is: Adapter(t)=Dropout(W2·(ReLU(W1·t))) Where Adapter(t) is the output of the adapter; t is the feature vector of the input adapter; W1 and W2 are the weight matrices of the two linear transformations of the adapter respectively; Dropout is the dropout operation.

7. The method for fine-tuning a large model for tutoring medical examinations based on small sample learning according to any one of claims 1 to 4 and 6, characterized in that: Before using the dataset to train the large model for medical examination tutoring, the weight information of the non-adapter part in the Transformer layer needs to be frozen, and only the weight of the adapter is updated during the training process.

8. The method for fine-tuning a large model for medical examination tutoring based on small sample learning according to claim 1, characterized in that: Methods for clustering word embedding vectors include: A1. Randomly select K cluster centers from all word embedding vectors and perform K-means clustering: Among them, μ j The jth cluster center selected by K-means; c i is the word embedding vector x i Index into the cluster; ||x i -μ j || 2 is the word embedding vector x i to μ j The square of the Euclidean distance; A2. Calculate the mean of all word embedding vectors in each cluster as the new cluster center: Among them, \C j \ is cluster C j The number of data points; is the new cluster center; A3: Determine whether the change between the current cluster center of each cluster and the cluster center of the last clustering is less than the preset threshold. If so, complete the clustering. Otherwise, update And return to step A1. The method of calculating the attention score of each word embedding vector and cluster center includes: B1. Calculate the cosine similarity between each word embedding vector and the cluster center: Among them, S ji is the word embedding vector x i and cluster C j The cluster center μ j The cosine similarity of ; ||·|| is the Euclidean norm of the vector; B2. Calculate the attention score of each word embedding vector based on cosine similarity: Among them, A ij For x i Relative to the cluster center μ j Attention score; S ki For x i and cluster center μ k The cosine similarity of , k is a variable, 1≤k≤K, and exp(.) is a natural exponential function.

9. The method for fine-tuning a large model for medical examination tutoring based on small sample learning according to claim 1, characterized in that: Methods for selecting word embedding vectors whose attention scores meet preset conditions as data sets include: C1. Sort the word embedding vectors in each cluster in descending order according to their attention scores, and select the word embedding vector with the highest attention score in each cluster; C2. Sort all selected word embedding vectors in descending order according to their attention scores, and select a preset number of word embedding vectors with the highest attention scores to form a data set.

10. A fine-tuning system for a large model of medical examination tutoring based on small sample learning, characterized in that: It includes a word embedding module, an oral test question small sample extraction component and a lightweight model fine-tuning component connected in sequence, wherein the lightweight model fine-tuning component includes a large model training module, a pruning and reasoning test module, an indicator calculation module, a judgment module and a threshold update module connected in sequence; The word embedding module is used to obtain oral knowledge test questions and convert the test questions into word embedding vectors using word embedding; The oral test small sample extraction component is used to cluster word embedding vectors and calculate the attention score between each word embedding vector and the cluster center, and select the word embedding vectors whose attention scores meet the preset conditions as the data set; The large model training module is used to train the medical examination tutoring large model using the data set and store the weights of the adapter during the training process, wherein the medical examination tutoring large model is composed of a Transformer layer embedded with an adapter; The pruning and reasoning test module is used to prune the weights of the adapter according to the importance index of the weights and the pruning threshold, and then perform reasoning tests on the medical examination tutoring model before and after pruning to obtain the reasoning time and test scores before and after pruning; The indicator calculation module is used to calculate the accuracy drop index and acceleration performance index of the medical examination tutoring model before and after pruning based on the inference time and test score before and after pruning; The judgment module is used to judge whether the accuracy drop index and the acceleration performance index or the number of pruning times meet the corresponding preset conditions. If so, a fine-tuned medical examination tutoring model is obtained, otherwise, the threshold update module is entered; The threshold update module is used to update the pruning threshold according to the accuracy drop index and the acceleration performance index, and then return to the pruning and reasoning test module.

Images