Methods, devices, electronic equipment and media for dynamic early termination of pre-trained models
By utilizing embedding and transformation layers in the pre-trained model for vector transformation and feature extraction, calculating the similarity of probability distributions of adjacent transformation layers, and determining whether to terminate early based on patience values, the problem of wasted computational resources and stringent termination criteria in large pre-trained models is solved, achieving efficient dynamic early termination and accurate prediction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2023-04-18
- Publication Date
- 2026-05-26
Smart Images
Figure CN116451152B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent decision-making, and in particular to a method, apparatus, electronic device, and readable storage medium for dynamic early termination of a pre-trained model. Background Technology
[0002] Early model termination refers to the process of ending the calculation of a model ahead of time. For example, in the financial industry, when classifying various insurance documents, due to the special nature of some insurance documents, an early model termination mechanism can be used to reduce resource consumption when classifying some insurance documents.
[0003] Most large pre-trained models currently suffer from excessive parameters and overthinking during training and application, wasting a lot of computing resources and increasing the difficulty of actual deployment and application. In addition, existing methods for dynamic early termination of pre-trained models generally use cross-layer comparison of the model as the standard for judging whether the model should terminate early. The exit criteria are too strict, making it difficult for the model to exit and resulting in low efficiency of early termination of pre-trained models. Summary of the Invention
[0004] This invention provides a method, apparatus, electronic device, and readable storage medium for dynamic early termination of pre-trained models, with the aim of improving the efficiency of dynamic early termination of pre-trained models.
[0005] To achieve the above objectives, the present invention provides a method for dynamic early termination of a pre-trained model, the method comprising:
[0006] Obtain training text and target text, use the training text to train a preset initial model, and obtain a pre-trained model that has been trained.
[0007] The target text is transformed into a text vector sequence using the embedding layer in the pre-trained model.
[0008] The text vector sequence is sequentially processed by the transformation layer in the pre-trained model to extract features, thereby obtaining a text feature vector.
[0009] The probability distribution of the target text category corresponding to the text feature vector is calculated sequentially using each classification layer connected to the conversion layer in the preset model;
[0010] Calculate the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers;
[0011] Based on the similarity, the patience value of each adjacent conversion layer is determined, and the patience values of the adjacent conversion layers are accumulated sequentially to obtain the target patience value;
[0012] When the target patience value meets the preset exit patience value, feature extraction of the text vector sequence is stopped, and the probability distribution calculated by the classification layer connected to the transformation layer where the text vector sequence stopped feature extraction is used as the probability distribution of the target text category.
[0013] Optionally, determining the patience value for each adjacent conversion layer based on the similarity includes:
[0014] Determine whether the similarity of the probability distributions in the classification layers corresponding to the adjacent conversion layers is less than a preset threshold;
[0015] When the similarity of the probability distribution in the classification layer corresponding to the adjacent conversion layer is not less than a preset threshold, the patience value of the adjacent conversion layer is set to zero.
[0016] When the similarity of the probability distribution in the classification layer corresponding to the adjacent conversion layer is less than a preset threshold, the patience value of the adjacent conversion layer is set to 1.
[0017] Optionally, calculating the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers includes:
[0018] Based on the probability distribution of the classification layers corresponding to the adjacent transformation layers, a predicted category histogram is generated;
[0019] The predicted category histogram is normalized to obtain a normalized histogram;
[0020] The normalized histogram is calculated using a preset Bach coefficient algorithm to obtain the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers.
[0021] Optionally, the sequence utilizes each classification layer connected to the transformation layer in the preset model to calculate the probability distribution of the target text category corresponding to the text feature vector, including:
[0022] The text feature vector is linearly transformed using the fourth, fifth, and sixth preset parameters in the classification layer to obtain the query vector, key vector, and numerical vector.
[0023] The similarity matrix is obtained by multiplying the query vector with the transpose of the key vector.
[0024] The similarity matrix is normalized to obtain a normalized matrix;
[0025] The normalization matrix is calculated using a preset activation function to obtain the activation matrix;
[0026] Multiplying the activation matrix by the numerical vector yields the probability distribution of the target text type corresponding to the text feature vector.
[0027] Optionally, the sequence utilizes the transformation layer in the pre-trained model to extract features from the text vector sequence to obtain a text feature vector, including:
[0028] The text vector sequence is encoded using the encoding module in the conversion layer to obtain a text encoded vector;
[0029] The text encoding vector is linearly transformed using the first, second and third preset parameters in the attention mechanism module of the transformation layer to obtain the text query vector, text key vector and text numerical vector.
[0030] The text weight vector is obtained by calculating the text query vector, the text key vector, and the text numerical vector using the normalized exponential function in the attention mechanism module.
[0031] The attention weight vector is decoded using the decoding module in the transformation layer to obtain the text feature vector.
[0032] Optionally, the step of using the embedding layer in the pre-trained model to perform vector transformation on the target text to obtain a text vector sequence includes:
[0033] The target text is segmented into words to obtain segmented text;
[0034] Extract word groups from the segmented text and construct a text dictionary based on the word groups;
[0035] Based on the sentence order of the target text and the word order in the text dictionary, a sentence vector is generated for each sentence in the target text;
[0036] The statement vectors are concatenated to obtain a text vector sequence.
[0037] Optionally, the step of training a preset initial model using the training text to obtain a pre-trained model includes:
[0038] Extract the original text types from the training text;
[0039] The training text is transformed into a vector sequence using the embedding layer in the preset initial model;
[0040] The training text vector sequence is sequentially processed by the transformation layer in the preset initial model to extract features, resulting in training text feature vectors.
[0041] The probability distribution of the training text category corresponding to the training text transformation vector is calculated sequentially using each classification layer connected to the transformation layer in the preset initial model;
[0042] Based on the probability distribution, determine the predicted text type of the training text;
[0043] Based on the original text type and the preset loss function, calculate the loss value for the predicted text type;
[0044] Based on the loss value, adjust the parameters in the preset initial model, and return to the step of using the embedding layer in the preset initial model to perform vector transformation on the training text to obtain a training text vector sequence, until the loss value meets the preset threshold, and obtain the pre-trained model that has been trained.
[0045] To address the above problems, the present invention also provides a pre-trained model dynamic early termination device, the device comprising:
[0046] The model training module is used to acquire training text and target text, and to train a preset initial model using the training text to obtain a pre-trained model.
[0047] The similarity calculation module is used to transform the target text into a vector using the embedding layer in the pre-trained model to obtain a text vector sequence, extract features from the text vector sequence using the transformation layer in the pre-trained model to obtain text feature vectors, calculate the probability distribution of the target text category corresponding to the text feature vector using each classification layer connected to the transformation layer in the preset model, and calculate the similarity of the probability distribution in the classification layers corresponding to adjacent transformation layers.
[0048] The model early termination prediction module is used to determine the patience value of each adjacent conversion layer based on the similarity, and to accumulate the patience values of the adjacent conversion layers in sequence to obtain the target patience value. When the target patience value meets the preset termination patience value, feature extraction of the text vector sequence is stopped, and the probability distribution calculated by the classification layer connected to the conversion layer where the text vector sequence stopped feature extraction is used as the probability distribution of the target text category.
[0049] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:
[0050] Memory, storing at least one computer program; and
[0051] The processor executes the computer program stored in the memory to implement the pre-trained model dynamic early termination method described above.
[0052] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the aforementioned pre-trained model dynamic early termination method.
[0053] This invention, through obtaining training text and target text, trains a pre-trained model using the training text, ensuring its usability. Next, it sequentially extracts features from the text vector sequence using the transformation layers in the pre-trained model, obtaining text feature vectors. Then, it sequentially calculates the probability distribution of the target text category corresponding to each text feature vector using each classification layer connected to the transformation layer in the pre-trained model, achieving cross-layer comparison and laying the groundwork for dynamic early termination of the pre-trained model. Finally, it determines the patience value of each adjacent transformation layer based on the similarity, and accumulates these patience values sequentially to obtain the target patience value. When the target patience value meets the preset termination patience value, feature extraction from the text vector sequence stops, and the probability distribution calculated by the classification layer connected to the transformation layer where feature extraction stopped is used as the probability distribution of the target text category. This simplifies the criteria for dynamic early termination of the pre-trained model, thereby improving its efficiency. Therefore, the present invention provides a method, apparatus, device, and storage medium for dynamic early termination of a pre-trained model, which can improve the efficiency of dynamic early termination of the pre-trained model. Attached Figure Description
[0054] Figure 1 This is a flowchart illustrating a pre-trained model dynamic early termination method provided in an embodiment of the present invention.
[0055] Figures 2 to 3 This is a detailed flowchart of one step in the pre-trained model dynamic early termination method provided in an embodiment of the present invention;
[0056] Figure 4 This is a schematic diagram of a module of a pre-trained model dynamic early termination device provided in an embodiment of the present invention;
[0057] Figure 5 This is a schematic diagram of the internal structure of an electronic device that implements a dynamic early termination method for a pre-trained model according to an embodiment of the present invention;
[0058] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0059] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0060] This invention provides a method for dynamic early termination of a pre-trained model. The execution entity of this method includes, but is not limited to, at least one of the following: a server, a terminal, or other electronic devices configured to execute the method provided in this application. In other words, the method can be executed by software or hardware installed on a terminal device or a server device, where the software may be a blockchain platform. The server may include an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0061] Reference Figure 1 The flowchart shown is a schematic diagram of a pre-trained model dynamic early termination method provided in an embodiment of the present invention. In this embodiment, the pre-trained model dynamic early termination method includes:
[0062] S1. Obtain training text and target text, and use the training text to train a preset initial model to obtain a pre-trained model.
[0063] In this embodiment of the invention, the training text can be text data of known text types used to train the preset initial model. The target text can be text data of the text type to be detected. The preset initial model can be a neural network model for detecting text types, generally including an embedding layer, a transformation layer, and a classification layer.
[0064] In this embodiment of the invention, the training text and the target text can be texts such as bill texts and insurance product descriptions in the financial field.
[0065] In optional embodiments of the present invention, training texts of known text types and target texts of unknown text types can be crawled by web crawlers, or the text data can be retrieved by querying the enterprise's internal text database, thereby ensuring the smooth acquisition and accuracy of training texts and target texts.
[0066] In this embodiment of the invention, the training text is used to train a preset initial model to obtain a pre-trained model that has been trained, ensuring that the pre-trained model can be used in subsequent practical applications, thereby realizing the dynamic early termination of the pre-trained model.
[0067] Furthermore, as an optional embodiment of the present invention, reference is made to... Figure 2 As shown, the step S1, which involves training a pre-set initial model using the training text to obtain a pre-trained model, includes:
[0068] S11. Extract the original text types of the training text;
[0069] S12. The training text is transformed into a vector using the embedding layer in the preset initial model to obtain a training text vector sequence.
[0070] S13. Sequentially use the transformation layer in the preset initial model to extract features from the training text vector sequence to obtain the training text feature vector;
[0071] S14. Sequentially calculate the probability distribution of the training text conversion vector corresponding to the training text category using each classification layer connected to the conversion layer in the preset initial model;
[0072] S15. Determine the predicted text type of the training text based on the probability distribution;
[0073] S16. Calculate the loss value of the predicted text type based on the original text type and the preset loss function;
[0074] S17. Adjust the parameters in the preset initial model according to the loss value, and return to the step of using the embedding layer in the preset initial model to perform vector transformation on the training text to obtain a training text vector sequence, until the loss value meets the preset threshold, and obtain the pre-trained model that has been trained.
[0075] In this embodiment of the invention, the preset threshold can be a value calculated by professionals based on actual results, which can ensure both model accuracy and prevent model overfitting.
[0076] In an optional embodiment of the present invention, the parameters of the preset initial model are adjusted by encoding and decoding the training text and predicting the text type, thereby obtaining a pre-trained model that has been trained, ensuring the accuracy of the pre-trained model, and thus ensuring the accuracy of the prediction results of the pre-trained model when it fails early.
[0077] S2. The target text is transformed into a vector using the embedding layer in the pre-trained model to obtain a text vector sequence.
[0078] In this embodiment of the invention, the embedding layer may be a module in the pre-trained model used for vectorizing text data.
[0079] In an optional embodiment of the present invention, since the pre-trained model can only accept vectorized data, the target text needs to be vectorized before predicting the text type of the target text.
[0080] In this embodiment of the invention, the embedding layer in the pre-trained model is used to transform the target text into a vector sequence, which ensures that the target text can smoothly enter the next module in the pre-trained model, thereby ensuring the continuity of the text category prediction.
[0081] Furthermore, as an optional embodiment of the present invention, S2 includes:
[0082] The target text is segmented into words to obtain segmented text;
[0083] Extract word groups from the segmented text and construct a text dictionary based on the word groups;
[0084] Based on the word groups contained in the text statements in the target text and the word groups in the text dictionary, generate a statement vector for each statement in the target text;
[0085] The statement vectors are concatenated to obtain a text vector sequence.
[0086] In this embodiment of the invention, the phrase may be a word or a single character.
[0087] In this embodiment of the invention, a text dictionary is obtained by arranging the phrases in the target text, and then a vector of the target text is generated based on the target text and the text dictionary, thereby realizing the vectorization of the target text.
[0088] In an optional embodiment of the present invention, for the target text "I like to take the high-speed rail home" and "China's high-speed rail is very fast", the two sentences are segmented into words to obtain the phrases "I, like, take, high-speed rail, home, China, high-speed rail, very, fast". The phrases are then de-duplicated and rearranged to obtain the text.
[0089] "China": 6, "Very": 7, "Fast": 8}. Finally, based on the sentence order of the target text and the word order in the text dictionary, a sentence vector is generated for each sentence in the target text, namely, the sentence vector [1,1,1,1,1,0,0,0] corresponding to the sentence "I like to take the high-speed rail home" and the sentence vector [0,0,0,1,0,1,1,1] corresponding to the sentence "China's high-speed rail is very fast". The i-th element in the sentence vector indicates whether the i-th word in the dictionary appears in the sentence.
[0090] S3. Sequentially use the transformation layer in the pre-trained model to extract features from the text vector sequence to obtain text feature vectors.
[0091] In this embodiment of the invention, the conversion layer may be a module that encodes and decodes the input vector.
[0092] In an optional embodiment of the present invention, since some text data is quite complex, the pre-trained model generally includes multiple transformation layers, with each transformation layer connected to a corresponding classification layer. The transformation layer can be a Transformer.
[0093] In this embodiment of the invention, the transformation layer in the pre-trained model is used to extract features from the text vector sequence to obtain text feature vectors. This determines the attention allocation for each sentence in the target text, thereby reducing the computation of the target text by the subsequent transformation layer and realizing the dynamic early termination of the pre-trained model.
[0094] Furthermore, as an optional embodiment of the present invention, S3 includes:
[0095] The text vector sequence is encoded using the encoding module in the conversion layer to obtain a text encoded vector;
[0096] The text encoding vector is linearly transformed using the first, second, and third preset parameters in the attention mechanism module of the transformation layer to obtain the text query vector, text key vector, and text numerical vector.
[0097] The text weight vector is obtained by calculating the text query vector, the text key vector, and the text numerical vector using the normalized exponential function in the attention mechanism module.
[0098] The attention weight vector is decoded using the decoding module in the transformation layer to obtain the text feature vector.
[0099] In this embodiment of the invention, the first, second and third preset parameters can be the transformation layer parameter matrix obtained through multiple training and optimization processes.
[0100] In an optional embodiment of the present invention, feature extraction of the text vector sequence is achieved through a series of actions such as encoding, attention weight calculation and decoding of the text vector sequence multiple times, thereby ensuring the accuracy of the text feature vector.
[0101] S4. Sequentially calculate the probability distribution of the target text category corresponding to the text feature vector using each classification layer connected to the conversion layer in the preset model.
[0102] In this embodiment, the classification layer may be a module that divides the target text into different text types.
[0103] In an optional embodiment of the present invention, since the text feature vector cannot directly reflect the text type of the target text, it is necessary to calculate the text feature vector to obtain the probability classification of the text type of the target text.
[0104] In this embodiment of the invention, the probability distribution of the target text category corresponding to the text feature vector is calculated sequentially using each classification layer connected to the conversion layer in the preset model, thereby achieving prediction of the target text category and ensuring that the accuracy of the prediction of the pre-trained model can be monitored layer by layer, thus achieving early model termination.
[0105] Furthermore, as an optional embodiment of the present invention, reference is made to... Figure 3 As shown, S4 includes:
[0106] S41. Using the fourth, fifth and sixth preset parameters in the classification layer, linear transformations are performed on the text feature vectors to obtain query vectors, key vectors and numerical vectors;
[0107] S42. Perform a dot product between the query vector and the transpose of the key vector to obtain a similarity matrix;
[0108] S43. Normalize the similarity matrix to obtain a normalized matrix;
[0109] S44. Calculate the normalization matrix using a preset activation function to obtain the activation matrix;
[0110] S45. Multiply the activation matrix by the numerical vector to obtain the probability distribution of the target text type corresponding to the text feature vector.
[0111] In this embodiment of the invention, the fourth, fifth, and sixth preset parameters can be classification layer parameter matrices obtained through multiple training and optimization processes. The preset activation function can be the softmax activation function.
[0112] In an optional embodiment of the present invention, by performing score normalization calculation on the similarity matrix, the matrix gradient becomes more stable and reliable, thereby improving the calculation efficiency of the probability distribution of the target text type corresponding to the text feature vector.
[0113] S5. Calculate the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers.
[0114] In an optional embodiment of the present invention, in order to reduce the consumption of model computing resources, when the predicted probability distribution in the classification layer is similar multiple times, the predicted category result of the target text can be determined to be the correct result, thereby stopping subsequent model calculations. This realizes the dynamic early termination of the pre-trained model and reduces the consumption of model computing resources.
[0115] Furthermore, in this embodiment of the invention, the similarity of the probability distributions in the classification layers corresponding to the adjacent transformation layers can be calculated by methods including but not limited to Kullback-Leibler divergence, distillation target, Hellinger distance, Bach distance, mutual information, Wasserstein distance, and cosine similarity.
[0116] In detail, as an optional embodiment of the present invention, S5 includes:
[0117] Based on the probability distribution of the classification layers corresponding to the adjacent transformation layers, a predicted category histogram is generated;
[0118] The predicted category histogram is normalized to obtain a normalized histogram;
[0119] The normalized histogram is calculated using a preset Bach coefficient algorithm to obtain the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers.
[0120] In detail, embodiments of the present invention can use the following formula to calculate the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers:
[0121]
[0122] Wherein, S can be the similarity of the probability distribution in the classification layers corresponding to adjacent transformation layers; N can be the number of text types; i can be the number in the text type sorting; p(i) can be the probability of the i-th text type in any layer of adjacent classification layers; p ′ (i) can be the probability of the i-th text type in another layer of the adjacent classification layer.
[0123] In an optional embodiment of the present invention, in order to simplify the calculation difficulty of the probability distribution similarity, the probability distribution can be converted into a histogram model first, thereby improving the calculation speed of probability distribution similarity and thus improving the efficiency of dynamic early termination of the pre-trained model.
[0124] S6. Determine the patience value of each adjacent conversion layer based on the similarity, and accumulate the patience values of the adjacent conversion layers in sequence to obtain the target patience value.
[0125] In this embodiment of the invention, the patience value can be a standard value used to determine whether it is possible to leave early.
[0126] In an optional embodiment of the present invention, in order to reduce the consumption of model computing resources and quickly achieve accurate prediction of the target text type, it is necessary to monitor the model for early termination. Therefore, it is necessary to use patience value to determine whether the model meets the early termination standard.
[0127] Further, as an optional embodiment of the present invention, determining the patience value of each adjacent conversion layer based on the similarity includes:
[0128] Determine whether the similarity of the probability distributions in the classification layers corresponding to the adjacent conversion layers is less than a preset threshold;
[0129] When the similarity of the probability distribution in the classification layer corresponding to the adjacent conversion layer is not less than a preset threshold, the patience value of the adjacent conversion layer is set to zero.
[0130] When the similarity of the probability distribution in the classification layer corresponding to the adjacent conversion layer is less than a preset threshold, the patience value of the adjacent conversion layer is set to 1.
[0131] In this embodiment of the invention, to ensure the accuracy of the prediction results of the pre-trained model, if the similarity of the probability distribution in the classification layer corresponding to the adjacent conversion layer is not less than a preset threshold, the patience value needs to be recalculated, thereby maximizing the accuracy of the prediction results of the pre-trained model.
[0132] S7. When the target patience value meets the preset exit patience value, stop feature extraction on the text vector sequence, and use the probability distribution calculated by the classification layer connected to the transformation layer where the text vector sequence stopped feature extraction as the probability distribution of the target text category.
[0133] In this embodiment of the invention, the preset exit patience value can be a standard value for early model termination calculated by professionals based on the model's accuracy.
[0134] In an optional embodiment of the present invention, when the target patience value meets the preset exit patience value, it indicates that the prediction result of the target text can represent the final prediction result of the pre-trained model for the target text. Therefore, when the target patience value meets the preset exit patience value, feature extraction of the text vector sequence can be stopped, and the probability distribution calculated by the classification layer connected to the transformation layer where the text vector sequence stops feature extraction is used as the probability distribution of the type of the target text.
[0135] This invention, through obtaining training text and target text, trains a pre-trained model using the training text, ensuring its usability. Next, it sequentially extracts features from the text vector sequence using the transformation layers in the pre-trained model, obtaining text feature vectors. Then, it sequentially calculates the probability distribution of the target text category corresponding to each text feature vector using each classification layer connected to the transformation layer in the pre-trained model, achieving cross-layer comparison and laying the groundwork for dynamic early termination of the pre-trained model. Finally, it determines the patience value of each adjacent transformation layer based on the similarity, and accumulates these patience values sequentially to obtain the target patience value. When the target patience value meets the preset termination patience value, feature extraction from the text vector sequence stops, and the probability distribution calculated by the classification layer connected to the transformation layer where feature extraction stopped is used as the probability distribution of the target text category. This simplifies the criteria for dynamic early termination of the pre-trained model, thereby improving its efficiency. Therefore, the present invention provides a method, apparatus, device, and storage medium for dynamic early termination of a pre-trained model, which can improve the efficiency of dynamic early termination of the pre-trained model.
[0136] like Figure 4 The diagram shown is a functional block diagram of the pre-trained model dynamic early termination device of the present invention.
[0137] The pre-trained model dynamic early termination device 100 of the present invention can be installed in an electronic device. Depending on the functions implemented, the pre-trained model dynamic early termination device 100 may include a model training module 101, a similarity calculation module 102, and a model early termination prediction module 103. The module mentioned in the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.
[0138] In this embodiment, the functions of each module / unit are as follows:
[0139] The model training module 101 is used to acquire training text and target text, and to train a preset initial model using the training text to obtain a pre-trained model.
[0140] In this embodiment of the invention, the training text can be text data of known text types used to train the preset initial model. The target text can be text data of the text type to be detected. The preset initial model can be a neural network model for detecting text types, generally including an embedding layer, a transformation layer, and a classification layer.
[0141] In this embodiment of the invention, the training text and the target text can be texts such as bill texts and insurance product descriptions in the financial field.
[0142] In optional embodiments of the present invention, training texts of known text types and target texts of unknown text types can be crawled by web crawlers, or the text data can be retrieved by querying the enterprise's internal text database, thereby ensuring the smooth acquisition and accuracy of training texts and target texts.
[0143] In this embodiment of the invention, the training text is used to train a preset initial model to obtain a pre-trained model that has been trained, ensuring that the pre-trained model can be used in subsequent practical applications, thereby realizing the dynamic early termination of the pre-trained model.
[0144] Furthermore, as an optional embodiment of the present invention, reference is made to... Figure 2 As shown, the step of training a preset initial model using the training text to obtain a pre-trained model includes:
[0145] Extract the original text types from the training text;
[0146] The training text is transformed into a vector sequence using the embedding layer in the preset initial model;
[0147] The training text vector sequence is sequentially processed by the transformation layer in the preset initial model to extract features, resulting in training text feature vectors.
[0148] The probability distribution of the training text category corresponding to the training text transformation vector is calculated sequentially using each classification layer connected to the transformation layer in the preset initial model;
[0149] Based on the probability distribution, determine the predicted text type of the training text;
[0150] Based on the original text type and the preset loss function, calculate the loss value for the predicted text type;
[0151] Based on the loss value, adjust the parameters in the preset initial model, and return to the step of using the embedding layer in the preset initial model to perform vector transformation on the training text to obtain a training text vector sequence, until the loss value meets the preset threshold, and obtain the pre-trained model that has been trained.
[0152] In this embodiment of the invention, the preset threshold can be a value calculated by professionals based on actual results, which can ensure both model accuracy and prevent model overfitting.
[0153] In an optional embodiment of the present invention, the parameters of the preset initial model are adjusted by encoding and decoding the training text and predicting the text type, thereby obtaining a pre-trained model that has been trained, ensuring the accuracy of the pre-trained model, and thus ensuring the accuracy of the prediction results of the pre-trained model when it fails early.
[0154] The similarity calculation module 102 is used to transform the target text into a vector using the embedding layer in the pre-trained model to obtain a text vector sequence, sequentially extract features from the text vector sequence using the conversion layer in the pre-trained model to obtain text feature vectors, sequentially calculate the probability distribution of the target text category corresponding to the text feature vector using each classification layer connected to the conversion layer in the preset model, and calculate the similarity of the probability distribution in the classification layer corresponding to adjacent conversion layers.
[0155] In this embodiment of the invention, the embedding layer may be a module in the pre-trained model used for vectorizing text data.
[0156] In an optional embodiment of the present invention, since the pre-trained model can only accept vectorized data, the target text needs to be vectorized before predicting the text type of the target text.
[0157] In this embodiment of the invention, the embedding layer in the pre-trained model is used to transform the target text into a vector sequence, which ensures that the target text can smoothly enter the next module in the pre-trained model, thereby ensuring the continuity of the text category prediction.
[0158] Furthermore, as an optional embodiment of the present invention, S2 includes:
[0159] The target text is segmented into words to obtain segmented text;
[0160] Extract word groups from the segmented text and construct a text dictionary based on the word groups;
[0161] Based on the word groups contained in the text statements in the target text and the word groups in the text dictionary, generate a statement vector for each statement in the target text;
[0162] The statement vectors are concatenated to obtain a text vector sequence.
[0163] In this embodiment of the invention, the phrase may be a word or a single character.
[0164] In this embodiment of the invention, a text dictionary is obtained by arranging the phrases in the target text, and then a vector of the target text is generated based on the target text and the text dictionary, thereby realizing the vectorization of the target text.
[0165] In an optional embodiment of the present invention, for the target text "I like to take the high-speed rail home" and "China's high-speed rail is very fast", the two sentences are segmented into words to obtain the phrases "I, like, sit, high-speed rail, home, China, high-speed rail, very, fast". The phrases are then deduplicated and rearranged to obtain a text dictionary {"I": 1, "like": 2, "sit": 3, "high-speed rail": 4, "home": 5, "China": 6, "very": 7, "fast": 8}. Finally, according to the sentence order of the target text and the phrase order in the text dictionary, a sentence vector is generated for each sentence in the target text, namely, the sentence vector corresponding to the sentence "I like to take the high-speed rail home" [1,1,1,1,1,0,0,0] and the sentence vector corresponding to the sentence "China's high-speed rail is very fast" [0,0,0,1,0,1,1,1]. The i-th element in the sentence vector indicates whether the i-th word in the dictionary appears in the sentence.
[0166] In this embodiment of the invention, the conversion layer may be a module that encodes and decodes the input vector.
[0167] In an optional embodiment of the present invention, since some text data is quite complex, the pre-trained model generally includes multiple transformation layers, with each transformation layer connected to a corresponding classification layer. The transformation layer can be a Transformer.
[0168] In this embodiment of the invention, the transformation layer in the pre-trained model is used to extract features from the text vector sequence to obtain text feature vectors. This determines the attention allocation for each sentence in the target text, thereby reducing the computation of the target text by the subsequent transformation layer and realizing the dynamic early termination of the pre-trained model.
[0169] Further, as an optional embodiment of the present invention, the sequential extraction of features from the text vector sequence using the transformation layer in the pre-trained model to obtain a text feature vector includes:
[0170] The text vector sequence is encoded using the encoding module in the conversion layer to obtain a text encoded vector;
[0171] The text encoding vector is linearly transformed using the first, second, and third preset parameters in the attention mechanism module of the transformation layer to obtain the text query vector, text key vector, and text numerical vector.
[0172] The text weight vector is obtained by calculating the text query vector, the text key vector, and the text numerical vector using the normalized exponential function in the attention mechanism module.
[0173] The attention weight vector is decoded using the decoding module in the transformation layer to obtain the text feature vector.
[0174] In this embodiment of the invention, the first, second and third preset parameters can be the transformation layer parameter matrix obtained through multiple training and optimization processes.
[0175] In an optional embodiment of the present invention, feature extraction of the text vector sequence is achieved through a series of actions such as encoding, attention weight calculation and decoding of the text vector sequence multiple times, thereby ensuring the accuracy of the text feature vector.
[0176] In this embodiment, the classification layer may be a module that divides the target text into different text types.
[0177] In an optional embodiment of the present invention, since the text feature vector cannot directly reflect the text type of the target text, it is necessary to calculate the text feature vector to obtain the probability classification of the text type of the target text.
[0178] In this embodiment of the invention, the probability distribution of the target text category corresponding to the text feature vector is calculated sequentially using each classification layer connected to the conversion layer in the preset model, thereby achieving prediction of the target text category and ensuring that the accuracy of the prediction of the pre-trained model can be monitored layer by layer, thus achieving early model termination.
[0179] Furthermore, as an optional embodiment of the present invention, reference is made to... Figure 3 As shown, the sequence utilizes each classification layer connected to the transformation layer in the preset model to calculate the probability distribution of the target text category corresponding to the text feature vector, including:
[0180] The text feature vector is linearly transformed using the fourth, fifth, and sixth preset parameters in the classification layer to obtain the query vector, key vector, and numerical vector.
[0181] The similarity matrix is obtained by multiplying the query vector with the transpose of the key vector.
[0182] The similarity matrix is normalized to obtain a normalized matrix;
[0183] The normalization matrix is calculated using a preset activation function to obtain the activation matrix;
[0184] Multiplying the activation matrix by the numerical vector yields the probability distribution of the target text type corresponding to the text feature vector.
[0185] In this embodiment of the invention, the fourth, fifth, and sixth preset parameters can be classification layer parameter matrices obtained through multiple training and optimization processes. The preset activation function can be the softmax activation function.
[0186] In an optional embodiment of the present invention, by performing score normalization calculation on the similarity matrix, the matrix gradient becomes more stable and reliable, thereby improving the calculation efficiency of the probability distribution of the target text type corresponding to the text feature vector.
[0187] In an optional embodiment of the present invention, in order to reduce the consumption of model computing resources, when the predicted probability distribution in the classification layer is similar multiple times, the predicted category result of the target text can be determined to be the correct result, thereby stopping subsequent model calculations. This realizes the dynamic early termination of the pre-trained model and reduces the consumption of model computing resources.
[0188] Furthermore, in this embodiment of the invention, the similarity of the probability distributions in the classification layers corresponding to the adjacent transformation layers can be calculated by methods including but not limited to Kullback-Leibler divergence, distillation target, Hellinger distance, Bach distance, mutual information, Wasserstein distance, and cosine similarity.
[0189] In detail, as an optional embodiment of the present invention, calculating the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers includes:
[0190] Based on the probability distribution of the classification layers corresponding to the adjacent transformation layers, a predicted category histogram is generated;
[0191] The predicted category histogram is normalized to obtain a normalized histogram;
[0192] The normalized histogram is calculated using a preset Bach coefficient algorithm to obtain the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers.
[0193] In detail, embodiments of the present invention can use the following formula to calculate the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers:
[0194]
[0195] Wherein, S can be the similarity of the probability distribution in the classification layers corresponding to adjacent transformation layers; N can be the number of text types; i can be the number in the text type sorting; p(i) can be the probability of the i-th text type in any layer of adjacent classification layers; p ′ (i) can be the probability of the i-th text type in another layer of the adjacent classification layer.
[0196] In an optional embodiment of the present invention, in order to simplify the calculation difficulty of the probability distribution similarity, the probability distribution can be converted into a histogram model first, thereby improving the calculation speed of probability distribution similarity and thus improving the efficiency of dynamic early termination of the pre-trained model.
[0197] The model early termination prediction module 103 is used to determine the patience value of each adjacent conversion layer based on the similarity, and to accumulate the patience values of the adjacent conversion layers in sequence to obtain the target patience value. When the target patience value meets the preset exit patience value, the feature extraction of the text vector sequence is stopped, and the probability distribution calculated by the classification layer connected to the conversion layer where the text vector sequence stopped feature extraction is used as the probability distribution of the target text category.
[0198] In this embodiment of the invention, the patience value can be a standard value used to determine whether it is possible to leave early.
[0199] In an optional embodiment of the present invention, in order to reduce the consumption of model computing resources and quickly achieve accurate prediction of the target text type, it is necessary to monitor the model for early termination. Therefore, it is necessary to use patience value to determine whether the model meets the early termination standard.
[0200] Further, as an optional embodiment of the present invention, determining the patience value of each adjacent conversion layer based on the similarity includes:
[0201] Determine whether the similarity of the probability distributions in the classification layers corresponding to the adjacent conversion layers is less than a preset threshold;
[0202] When the similarity of the probability distribution in the classification layer corresponding to the adjacent conversion layer is not less than a preset threshold, the patience value of the adjacent conversion layer is set to zero.
[0203] When the similarity of the probability distribution in the classification layer corresponding to the adjacent conversion layer is less than a preset threshold, the patience value of the adjacent conversion layer is set to 1.
[0204] In this embodiment of the invention, to ensure the accuracy of the prediction results of the pre-trained model, if the similarity of the probability distribution in the classification layer corresponding to the adjacent conversion layer is not less than a preset threshold, the patience value needs to be recalculated, thereby maximizing the accuracy of the prediction results of the pre-trained model.
[0205] In this embodiment of the invention, the preset exit patience value can be a standard value for early model termination calculated by professionals based on the model's accuracy.
[0206] In an optional embodiment of the present invention, when the target patience value meets the preset exit patience value, it indicates that the prediction result of the target text can represent the final prediction result of the pre-trained model for the target text. Therefore, when the target patience value meets the preset exit patience value, feature extraction of the text vector sequence can be stopped, and the probability distribution calculated by the classification layer connected to the transformation layer where the text vector sequence stops feature extraction is used as the probability distribution of the type of the target text.
[0207] like Figure 5 The diagram shown is a schematic representation of the electronic device that implements the dynamic early termination method for pre-trained models according to the present invention.
[0208] The electronic device may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a pre-trained model dynamic early termination program.
[0209] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a pre-trained model's dynamic early termination program, but also to temporarily store data that has been output or will be output.
[0210] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., pre-trained model dynamic early termination programs) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.
[0211] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The communication bus 12 is configured to enable communication between the memory 11 and at least one processor 10, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus.
[0212] Figure 5 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 5 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.
[0213] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.
[0214] Optionally, the communication interface 13 may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device and other electronic devices.
[0215] Optionally, the communication interface 13 may further include a user interface, which may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.
[0216] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.
[0217] The pre-trained model dynamic early termination program stored in the memory 11 of the electronic device is a combination of multiple computer programs. When run in the processor 10, it can achieve the following:
[0218] Obtain training text and target text, use the training text to train a preset initial model, and obtain a pre-trained model that has been trained.
[0219] The target text is transformed into a text vector sequence using the embedding layer in the pre-trained model.
[0220] The text vector sequence is sequentially processed by the transformation layer in the pre-trained model to extract features, thereby obtaining a text feature vector.
[0221] The probability distribution of the target text category corresponding to the text feature vector is calculated sequentially using each classification layer connected to the conversion layer in the preset model;
[0222] Calculate the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers;
[0223] Based on the similarity, the patience value of each adjacent conversion layer is determined, and the patience values of the adjacent conversion layers are accumulated sequentially to obtain the target patience value;
[0224] When the target patience value meets the preset exit patience value, feature extraction of the text vector sequence is stopped, and the probability distribution calculated by the classification layer connected to the transformation layer where the text vector sequence stopped feature extraction is used as the probability distribution of the target text category.
[0225] Specifically, the processor 10's implementation method of the above-mentioned computer program can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.
[0226] Furthermore, if the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or volatile. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).
[0227] Embodiments of the present invention may also provide a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:
[0228] Obtain training text and target text, use the training text to train a preset initial model, and obtain a pre-trained model that has been trained.
[0229] The target text is transformed into a text vector sequence using the embedding layer in the pre-trained model.
[0230] The text vector sequence is sequentially processed by the transformation layer in the pre-trained model to extract features, thereby obtaining a text feature vector.
[0231] The probability distribution of the target text category corresponding to the text feature vector is calculated sequentially using each classification layer connected to the conversion layer in the preset model;
[0232] Calculate the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers;
[0233] Based on the similarity, the patience value of each adjacent conversion layer is determined, and the patience values of the adjacent conversion layers are accumulated sequentially to obtain the target patience value;
[0234] When the target patience value meets the preset exit patience value, feature extraction of the text vector sequence is stopped, and the probability distribution calculated by the classification layer connected to the transformation layer where the text vector sequence stopped feature extraction is used as the probability distribution of the target text category.
[0235] Furthermore, the computer's usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system, applications required for at least one function, etc.; and the data storage area may store data created based on the use of blockchain nodes, etc.
[0236] In the several embodiments provided by this invention, it should be understood that the disclosed electronic devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0237] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0238] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0239] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0240] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.
[0241] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.
[0242] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.
[0243] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims
1. A method for dynamic early termination of a pre-trained model, characterized in that, The method includes: Obtain training text and target text, use the training text to train a preset initial model, and obtain a pre-trained model that has been trained. The target text is transformed into a text vector sequence using the embedding layer in the pre-trained model. The text vector sequence is sequentially processed by the transformation layer in the pre-trained model to extract features, thereby obtaining a text feature vector. The probability distribution of the target text category corresponding to the text feature vector is calculated sequentially using each classification layer connected to the transformation layer in the pre-trained model; Calculate the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers; Based on the similarity, the patience value of each adjacent conversion layer is determined, and the patience values of the adjacent conversion layers are accumulated sequentially to obtain the target patience value; When the target patience value meets the preset exit patience value, feature extraction of the text vector sequence is stopped, and the probability distribution calculated by the classification layer connected to the transformation layer where the text vector sequence stopped feature extraction is used as the probability distribution of the target text category. The step of determining the patience value of each adjacent conversion layer based on the similarity includes: Determine whether the similarity of the probability distributions in the classification layers corresponding to the adjacent conversion layers is less than a preset threshold; When the similarity of the probability distribution in the classification layer corresponding to the adjacent conversion layer is not less than a preset threshold, the patience value of the adjacent conversion layer is set to zero. When the similarity of the probability distribution in the classification layer corresponding to the adjacent conversion layer is less than a preset threshold, the patience value of the adjacent conversion layer is set to 1.
2. The pre-trained model dynamic early termination method as described in claim 1, characterized in that, The calculation of the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers includes: Based on the probability distribution of the classification layers corresponding to the adjacent transformation layers, a predicted category histogram is generated; The predicted category histogram is normalized to obtain a normalized histogram; The normalized histogram is calculated using a preset Bach coefficient algorithm to obtain the similarity of the probability distributions in the classification layers corresponding to adjacent transformation layers.
3. The pre-trained model dynamic early termination method as described in claim 1, characterized in that, The sequence utilizes each classification layer connected to the transformation layer in the pre-trained model to calculate the probability distribution of the target text category corresponding to the text feature vector, including: The text feature vector is linearly transformed using the fourth, fifth, and sixth preset parameters in the classification layer to obtain the query vector, key vector, and numerical vector. The similarity matrix is obtained by multiplying the query vector with the transpose of the key vector. The similarity matrix is normalized to obtain a normalized matrix; The normalization matrix is calculated using a preset activation function to obtain the activation matrix; Multiplying the activation matrix by the numerical vector yields the probability distribution of the target text type corresponding to the text feature vector.
4. The pre-trained model dynamic early termination method as described in claim 1, characterized in that, The sequence utilizes the transformation layer in the pre-trained model to extract features from the text vector sequence, obtaining text feature vectors, including: The text vector sequence is encoded using the encoding module in the conversion layer to obtain a text encoded vector; The text encoding vector is linearly transformed using the first, second and third preset parameters in the attention mechanism module of the transformation layer to obtain the text query vector, text key vector and text numerical vector. The text weight vector is obtained by calculating the text query vector, the text key vector, and the text numerical vector using the normalized exponential function in the attention mechanism module. The text weight vector is decoded using the decoding module in the transformation layer to obtain the text feature vector.
5. The pre-trained model dynamic early termination method as described in claim 1, characterized in that, The step of using the embedding layer in the pre-trained model to perform vector transformation on the target text to obtain a text vector sequence includes: The target text is segmented into words to obtain segmented text; Extract word groups from the segmented text and construct a text dictionary based on the word groups; Based on the sentence order of the target text and the word order in the text dictionary, a sentence vector is generated for each sentence in the target text; The statement vectors are concatenated to obtain a text vector sequence.
6. The pre-trained model dynamic early termination method as described in claim 1, characterized in that, The step of training a preset initial model using the training text to obtain a pre-trained model includes: Extract the original text types from the training text; The training text is transformed into a vector sequence using the embedding layer in the preset initial model; The training text vector sequence is sequentially processed by the transformation layer in the preset initial model to extract features, resulting in training text feature vectors. The probability distribution of the training text category corresponding to the training text transformation vector is calculated sequentially using each classification layer connected to the transformation layer in the preset initial model; Based on the probability distribution, determine the predicted text type of the training text; Based on the original text type and the preset loss function, calculate the loss value for the predicted text type; Based on the loss value, adjust the parameters in the preset initial model, and return to the step of using the embedding layer in the preset initial model to perform vector transformation on the training text to obtain a training text vector sequence, until the loss value meets the preset threshold, and obtain the pre-trained model that has been trained.
7. A pre-trained model dynamic early termination device, used to implement the pre-trained model dynamic early termination method as described in any one of claims 1 to 6, characterized in that, The device includes: The model training module is used to acquire training text and target text, and to train a preset initial model using the training text to obtain a pre-trained model. The similarity calculation module is used to transform the target text into a vector using the embedding layer in the pre-trained model to obtain a text vector sequence, extract features from the text vector sequence using the transformation layer in the pre-trained model to obtain text feature vectors, calculate the probability distribution of the target text category corresponding to the text feature vector using each classification layer connected to the transformation layer in the pre-trained model, and calculate the similarity of the probability distribution in the classification layers corresponding to adjacent transformation layers. The model early termination prediction module is used to determine the patience value of each adjacent conversion layer based on the similarity, and to accumulate the patience values of the adjacent conversion layers in sequence to obtain the target patience value. When the target patience value meets the preset termination patience value, feature extraction of the text vector sequence is stopped, and the probability distribution calculated by the classification layer connected to the conversion layer where the text vector sequence stopped feature extraction is used as the probability distribution of the target text category.
8. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the pre-trained model dynamic early termination method as described in any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the pre-trained model dynamic early termination method as described in any one of claims 1 to 6.