Method and device for fine-tuning bert model, equipment and storage medium

Abstract

Embodiments of the present application provide a BERT model fine-tuning method and device, equipment and a storage medium, and relate to the technical field of computers. The method comprises: generating an original token list according to a text corpus, and obtaining word frequency statistical information of the original token list; determining a mask generation strategy based on the word frequency statistical information, and performing mask processing on the original token list according to the mask generation strategy to obtain a mask processing result; and performing iterative learning on a pre-trained BERT model based on the mask processing result until a preset condition is met, to obtain a fine-tuned BERT model. The present application statistically obtains word frequency information corresponding to a text corpus, and adaptively adjusts a mask generation probability according to the word frequency, so that the mask distribution generated by tokens of different frequencies is more uniform, and the semantic information of the text corpus is better preserved, thereby improving the learning ability of the BERT model for masked corpus.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method, apparatus, device, and storage medium for fine-tuning a BERT model. Background Technology

[0002] With the continuous development of natural language processing technology, pre-trained language models have performed exceptionally well in many tasks, with the BERT model being the most representative. The BERT model is a bidirectional pre-trained encoder that has achieved significant performance improvements in many natural language processing tasks, such as question answering, text classification, and named entity recognition.

[0003] Currently, when fine-tuning BERT using a small number of samples from a specific domain, the performance improvement of the BERT model is not significant because the static MASK mechanism of BERT makes it difficult to generate diverse masks, thus making it difficult to learn complex semantic representations. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, device, and storage medium for fine-tuning a BERT model, so as to improve the semantic learning performance of the BERT model for corpus samples in a specific domain.

[0005] In a first aspect, embodiments of this application provide a method for fine-tuning a BERT model, including:

[0006] Generate an original word list based on the text corpus, and obtain the word frequency statistics of the original word list;

[0007] Based on the word frequency statistics, a mask generation strategy is determined, and the original word list is masked according to the mask generation strategy to obtain the masking result.

[0008] Based on the masking results, the pre-trained BERT model is iteratively trained until the preset conditions are met, resulting in a fine-tuned BERT model.

[0009] In this embodiment, by statistically analyzing the word frequency information corresponding to the text corpus and adaptively adjusting the mask generation probability based on the word frequency, the mask distribution of word generation at different frequencies becomes more uniform, better preserving the semantic information of the text corpus, thereby improving the BERT model's ability to learn masked corpus.

[0010] In one possible embodiment, determining a masking strategy based on the word frequency statistics and masking the original word list according to the masking strategy to obtain a masking result includes:

[0011] Based on the word frequency statistics, the original word list is pre-sampled to obtain a pre-sampled word list.

[0012] The final generation probability is determined based on the preset mask generation probability and the comparison of the number of words in the original word list and the pre-sampling list.

[0013] The original word list is masked based on the pre-sampled word list and the final generation probability to obtain the masking result.

[0014] In this embodiment, the process of determining the mask generation probability based on word frequency statistics is divided into two stages, which further improves the uniformity of mask generation for words with different frequencies. Furthermore, based on the length of the corpus segment and the pre-sampling length, each corpus segment has an appropriate proportion of words that are masked, thereby further preserving the contextual information of the corpus and effectively improving the BERT model's ability to learn masked corpora.

[0015] In one possible embodiment, the step of pre-sampling the original word list based on the word frequency statistics to obtain a pre-sampled word list includes:

[0016] The frequency of occurrence of each word in the original word list is obtained based on the word frequency statistics information;

[0017] The probability of each word being extracted from the original word list is determined based on the frequency of occurrence of each word.

[0018] Based on the probability of each word being extracted, the original word list is pre-sampled to obtain a pre-sampled word list.

[0019] In this embodiment of the application, by performing frequency statistics on each word in the original word list and using it as the basis for calculating the probability of extraction, the uniformity of masking processing for words with different frequencies is further improved, effectively enhancing the BERT model's learning ability on masked corpora.

[0020] In one possible embodiment, determining the final generation probability based on a preset mask generation probability and a comparison of the number of tokens in the original token list and the pre-sampling list includes:

[0021] Obtain the target ratio of the number of lexical units in the original lexical unit list to the number of lexical units in the presampled list;

[0022] The final generation probability is determined by multiplying the preset mask generation probability by the target ratio.

[0023] In this embodiment, the final probability of each word in the word list being replaced with a mask is adaptively adjusted according to the ratio of the length of the word list to the length of the pre-sampling list, so that an appropriate proportion of words in each sentence are replaced with masks, thereby further preserving the contextual information of the text and effectively improving the BERT model's ability to learn masked text.

[0024] In one possible embodiment, determining a masking strategy based on the word frequency statistics and masking the original word list according to the masking strategy to obtain a masking result includes:

[0025] The frequency of occurrence of each word in the original word list is obtained based on the word frequency statistics information;

[0026] The probability of each word being extracted from the original word list is determined based on the frequency of occurrence of each word.

[0027] The original word list is masked based on the extraction probability of each word to obtain the masking result.

[0028] In this embodiment, the final probability of each word being replaced by a mask is determined directly based on the frequency of each word's occurrence, ensuring the uniformity of the distribution of words with different frequencies being generated into masks, and effectively improving the BERT model's ability to learn masked corpora.

[0029] In one possible embodiment, the sum of the probability of each word being extracted and the frequency of occurrence of the corresponding word is 1.

[0030] In the embodiments of this application, the sum of the probability of each word being extracted and the frequency of occurrence of each word is 1, so that low-frequency words are replaced with masks with a relatively high probability and high-frequency words are replaced with masks with a relatively low probability, ensuring the uniformity of the distribution of masks generated from words of different frequencies, and effectively improving the learning ability of the BERT model on masked corpora.

[0031] In one possible embodiment, the step of masking the original word list according to the masking generation strategy to obtain the masking result includes:

[0032] A list of terms to be processed is determined according to the mask generation strategy; wherein, the list of terms to be processed includes several mask terms that are determined to be masked and processed;

[0033] The mask position is determined according to the list of words to be processed, and the target words in the original word list corresponding to the mask position are masked to obtain the mask word list. The target words corresponding to the mask position are saved to the word tag list.

[0034] The word tag list and the mask word list are used as the masking processing result.

[0035] In this embodiment of the application, the masking result specifically includes a masked word list and a word tag list corresponding to the masked word. By providing multiple forms of masking results as the data basis for model training, the learning ability of the BERT model to masked corpora is improved.

[0036] In one possible embodiment, the iterative learning of the pre-trained BERT model based on the mask processing result until a preset condition is met, to obtain a fine-tuned BERT model, includes:

[0037] Based on the pre-trained BERT model, the prediction results corresponding to the masked word list are obtained, and based on the word label list, the true word label results are determined.

[0038] Obtain the cross-entropy loss between the predicted result and the actual word tag result;

[0039] The pre-trained BERT model is iteratively trained until the cross-entropy loss no longer decreases, resulting in a fine-tuned BERT model.

[0040] In the embodiments of this application, the specific fine-tuning process of the BERT model is guided by the cross-entropy loss between the prediction results and the real labels, which can accurately reflect the semantic awareness of the model, thereby improving the BERT model's ability to learn masked corpora.

[0041] In one possible embodiment, generating an original word list based on the text corpus and obtaining word frequency statistics of the original word list includes:

[0042] The acquired text corpus is cleaned, and the cleaned text corpus is then divided into several text corpus segments;

[0043] Each text corpus segment is divided into lexical units based on preset segmentation rules, resulting in an original list of lexical units corresponding to each text corpus segment.

[0044] In the embodiments of this application, the specific process of generating the original word list includes preprocessing operations such as data cleaning and segmentation to improve the accuracy of the input data, thereby further improving the BERT model's ability to learn from masked corpora.

[0045] Secondly, embodiments of this application provide a fine-tuning device for a BERT model, comprising:

[0046] The word frequency statistics module is used to generate an original word list based on the text corpus and obtain word frequency statistics information of the original word list;

[0047] The masking module is used to determine a masking generation strategy based on the word frequency statistics information, and to perform masking processing on the original word list according to the masking generation strategy to obtain the masking processing result.

[0048] The model fine-tuning module is used to iteratively learn the pre-trained BERT model based on the mask processing results until the preset conditions are met, thereby obtaining a fine-tuned BERT model.

[0049] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0050] Fourthly, embodiments of this application provide a computer program product, which includes a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0051] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method described in any embodiment of the first aspect. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart illustrating a fine-tuning method for a BERT model provided in an embodiment of this application;

[0054] Figure 2 This is a schematic diagram of the word frequency distribution of the training corpus provided in the embodiments of this application;

[0055] Figure 3 This is a schematic diagram of the masking operation process provided in an embodiment of this application;

[0056] Figure 4 A schematic diagram of the mask distribution of training corpus based on a static masking strategy provided for embodiments of this application;

[0057] Figure 5 A schematic diagram of the mask distribution of training corpus based on an adaptive dynamic masking strategy provided in an embodiment of this application;

[0058] Figure 6 A schematic diagram of a fine-tuning device for a BERT model provided in an embodiment of this application;

[0059] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0061] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0062] In existing technologies, when fine-tuning a pre-trained language model using a small amount of domain-specific corpus, the distribution of tokens (representing the smallest semantic unit, also known as a lexical unit in this scheme) in the domain-specific corpus becomes uneven due to the significantly smaller amount of such corpus compared to the general domain. If a static masking strategy is used in this situation, the uneven token distribution leads to a large number of high-frequency words being replaced with masks ([MASK] or other words), while only a small number of low-frequency words are masked, making it difficult for the BERT model to learn the complex semantic representations of domain-specific knowledge. Furthermore, using a dictionary-based masking strategy results in all industry-related tokens in a single training corpus being replaced with masks, thus failing to input industry-related context into the BERT model, further hindering its ability to learn the semantic representations of domain-specific knowledge.

[0063] In view of the problems existing in the above-mentioned prior art, such as Figure 1 As shown in the figure, this application embodiment provides a method for fine-tuning a BERT model, which may include the following steps:

[0064] S1. Generate an original word list based on the text corpus and obtain the word frequency statistics of the original word list;

[0065] S2. Determine the mask generation strategy based on word frequency statistics, and perform masking on the original word list according to the mask generation strategy to obtain the masking result;

[0066] S3. Based on the masking results, iteratively learn the pre-trained BERT model until the preset conditions are met, and obtain the fine-tuned BERT model.

[0067] First, a raw word list is generated based on the input text corpus. The text corpus can be an entire text or a single sentence. For an input text corpus, it can be divided into single sentences based on punctuation marks (periods, question marks, commas, parentheses, etc.), so that each raw word list corresponds to a single sentence. Then, the word frequency statistics for each raw word list are obtained. For example, the number of times each word appears in all texts is counted, and the total number of words in the text corpus is calculated. The word frequency of each word is then determined based on the ratio of the former to the latter.

[0068] After obtaining the word frequency statistics for each word list, a mask generation strategy can be determined based on the word frequency statistics. Specifically, the corresponding mask probability is calculated according to the word frequency of each word using preset rules, and then the original word list is masked based on the mask probability of each word to obtain the masking result.

[0069] Since each word in the existing technology is replaced with a mask with the same probability, in the final generation of masks, words that appear more frequently in the original corpus will be replaced with a large number of masks due to their large cardinality. Conversely, words that appear less frequently in the original corpus will be replaced with fewer or no masks due to their small cardinality. This results in the BERT model being unable to learn the semantic information corresponding to low-frequency words.

[0070] Unlike existing technologies, in the masking process of this application, lexical units with different frequencies will be replaced with masks with different probabilities, making the distribution of lexical units that are ultimately replaced with masks more uniform. This allows for better preservation of the semantic information of the text corpus, thereby improving the BERT model's ability to learn masked corpora.

[0071] In one possible embodiment, step S2 (determining a mask generation strategy based on word frequency statistics, and performing masking processing on the original word list according to the mask generation strategy to obtain the masking result) may include:

[0072] S201. Based on word frequency statistics, perform word presampling on the original word list to obtain a presampled word list;

[0073] S202. Determine the final generation probability based on the preset mask generation probability and the comparison of the number of words in the original word list and the pre-sampling list.

[0074] S203. Mask the original word list based on the pre-sampled word list and the final generation probability to obtain the masking result.

[0075] It should be noted that in practical applications, since the length of each text corpus is different, the number of words in each original word list is also different. If the mask generation strategy is determined solely by word frequency statistics, there may be a large proportion of words in some original word lists that are replaced with masks, while a small proportion of words in some original word lists are replaced with masks, resulting in an uneven probability distribution of the final mask words.

[0076] To address this, this application's embodiments further improve the uniformity of mask generation for words with different frequencies by dividing the process of determining the mask generation probability based on word frequency statistics into two stages. Specifically, in the first stage, the original word list is pre-sampled based on word frequency statistics. Then, in the second stage, the final generation probability of words in the pre-sampled list being replaced with masks is adaptively adjusted based on the length relationship between the original word list and the pre-sampled list. Through this two-stage adaptive adjustment of mask generation probability, not only is the uniformity of word distribution in the final generated mask guaranteed, but also an appropriate proportion of words in each corpus segment are masked, thereby further preserving the contextual information of the corpus and effectively improving the BERT model's learning ability for masked corpora.

[0077] In one possible embodiment, step S201 (pre-sampling the original word list based on word frequency statistics to obtain a pre-sampled word list) may include:

[0078] S2011. Obtain the frequency of occurrence of each word element in the original word element list based on word frequency statistics;

[0079] S2012. Determine the probability of each word element being extracted from the original word element list based on the frequency of each word element's occurrence.

[0080] S2013. Based on the probability of each word being extracted, perform word presampling on the original word list to obtain a presampled word list.

[0081] It should be noted that, in the embodiments of this application, by performing frequency statistics on each word in the original word list and using it as the basis for calculating the probability of extraction, the precision of word frequency statistics is improved, the uniformity of masking processing for words of different frequencies is further improved, and the learning ability of the BERT model on the masked corpus is effectively improved.

[0082] In one possible embodiment, S202 (determining the final generation probability based on a preset mask generation probability and a comparison of the number of words in the original word list and the pre-sampling list) may include:

[0083] S2021. Obtain the target ratio of the number of lexical units in the original lexical unit list to the number of lexical units in the presampled list;

[0084] S2022. The final generation probability is determined based on the product of the preset mask generation probability and the target ratio.

[0085] It should be noted that, to ensure that an appropriate proportion of words in the original word list corresponding to each text corpus are replaced with masks, a target ratio of the number of words in the original word list to the number of words in the pre-sampled list can be calculated. The final generation probability of each word being replaced with a mask is then determined by multiplying the target ratio by a preset mask generation probability. Understandably, the preset mask generation probability is typically set to 15%. In existing technical solutions, each word in the text corpus will be replaced with a mask with the same 15% probability.

[0086] In this embodiment, due to the first stage of lexical presampling, each lexical in each sentence of text corpus (each original lexical list) is added to the presampling list with different probabilities (determined according to the word frequency of each lexical). Therefore, the proportion of lexicals extracted from each original lexical list to the presampling list is different. In order to adaptively adjust the proportion of the number of lexicals in the final generated mask to the total number of lexicals in the original lexical list, it can be adjusted according to the ratio of the length of the original lexical list to the length of the corresponding presampling list. Understandably, if the ratio of the length of the original word list to the length of the corresponding pre-sampling list is too large, it indicates that the number of words in the pre-sampling list corresponding to the original word list is too small. It is necessary to use the product of the preset mask generation probability and this ratio as the final generation probability for masking the words in the pre-sampling list. Since the final generation probability is adjusted to be greater than the normal preset mask generation probability, more words can be extracted from the pre-sampling list for masking, so that the proportion of words in the original word list that are ultimately masked reaches a normal proportion, thereby making up for the problem of the small number of words in the pre-sampling list corresponding to the original word list.

[0087] In this embodiment, the final probability of each word in the word list being replaced with a mask is adaptively adjusted according to the ratio of the length of the word list to the length of the pre-sampling list, so that an appropriate proportion of words in each sentence are replaced with masks, thereby further preserving the contextual information of the text and effectively improving the BERT model's ability to learn masked text.

[0088] In one possible embodiment, S2 (determining a mask generation strategy based on word frequency statistics, and performing masking processing on the original word list according to the mask generation strategy to obtain the masking result) may further include:

[0089] S211. Obtain the frequency of occurrence of each word in the original word list based on word frequency statistics;

[0090] S212. Determine the probability of each word element being extracted from the original word element list based on the frequency of each word element's occurrence.

[0091] S213. Mask the original word list based on the extraction probability of each word to obtain the masking result.

[0092] In this embodiment, a two-stage mask generation strategy can be omitted. The final probability of each word being replaced by a mask can be determined directly based on its frequency of occurrence. Although the proportion of each word in the original word list that is replaced by a mask is not adaptively adjusted (equivalent to not considering the contextual information between sentences in the original corpus), the uniformity of the distribution of words with different frequencies being generated as masks is still guaranteed. Compared with the existing technology that masks each word in the text corpus with the same probability, it can still effectively improve the BERT model's learning ability for masked corpora.

[0093] In one possible embodiment, the sum of the probability of each word being extracted and the frequency of occurrence of the corresponding word is 1.

[0094] In the embodiments of this application, the sum of the probability of each word being extracted and the frequency of occurrence of each word is 1, so that low-frequency words are replaced with masks with a relatively high probability and high-frequency words are replaced with masks with a relatively low probability, ensuring the uniformity of the distribution of masks generated from words of different frequencies, and effectively improving the learning ability of the BERT model on masked corpora.

[0095] In one possible embodiment, the original word list is masked according to a masking generation strategy to obtain a masking result, including:

[0096] The list of words to be processed is determined based on the masking generation strategy; wherein, the list of words to be processed includes several masked words that are determined to be masked and processed;

[0097] Determine the mask position based on the list of words to be processed, and mask the target words in the original word list that correspond to the mask position to obtain the mask word list. Save the target words corresponding to the mask position to the word tag list.

[0098] The list of lexical tags and the list of masked lexical tags are used as the result of masking.

[0099] In this embodiment of the application, the masking result specifically includes a masked word list and a word tag list corresponding to the masked word. By providing multiple forms of masking results as the data basis for model training, the learning ability of the BERT model to masked corpora is improved.

[0100] In one possible embodiment, S3 (iteratively learning the pre-trained BERT model based on the masking results until a preset condition is met, thus obtaining a fine-tuned BERT model) may include:

[0101] The pre-trained BERT model is used to obtain the prediction results corresponding to the masked word list, and the true word label results are determined based on the word label list.

[0102] Obtain the cross-entropy loss between the predicted results and the actual word tag results;

[0103] The pre-trained BERT model is iteratively trained until the cross-entropy loss no longer decreases, resulting in a fine-tuned BERT model.

[0104] In the embodiments of this application, the specific fine-tuning process of the BERT model is guided by the cross-entropy loss between the prediction results and the real labels, which can accurately reflect the semantic awareness of the model, thereby improving the BERT model's ability to learn masked corpora.

[0105] In one possible embodiment, S1 (generating an original word list based on the text corpus and obtaining word frequency statistics of the original word list) may include:

[0106] The acquired text corpus is cleaned, and the cleaned text corpus is then divided into several text corpus segments;

[0107] Each text corpus segment is divided into lexical units based on preset segmentation rules, resulting in an original list of lexical units corresponding to each text corpus segment.

[0108] It should be noted that since the input corpus may often contain HTML tags, special characters, etc., data cleaning is necessary to prevent these contents from interfering with subsequent steps such as word frequency statistics. In the embodiments of this application, the specific process of generating the original word list includes preprocessing operations such as data cleaning and segmentation to improve the accuracy of the input data, thereby further improving the BERT model's ability to learn from masked corpora.

[0109] As an example, the implementation process of the embodiments of this application will be described in detail below, which may include the following steps:

[0110] 1. Data preprocessing:

[0111] Load the BERT model (an open-source model that is commercially viable and modifiable) pre-trained on a general corpus, including the BERT dictionary vocab.txt, the BERT model structure, and the BERT model weights. Acquire domain-specific text corpora and clean the text data, such as removing HTML tags and special characters. Next, use the WordPiece algorithm for word segmentation, dividing the text into continuous subunits called tokens (the smallest semantic unit, i.e., lexical units; it should be understood that tokens are simply the plural form of token, and in this description, "tokens" and "token" essentially refer to the same concept, i.e., "lexical unit").

[0112] For example, the word "unbelievable" can be segmented into three tokens: "un", "##believ", and "##able". After segmentation, special markers (including [CLS] and [SEP]) are added to each corpus (using punctuation marks as the dividing line between corpora). Since BERT requires all input sequences to be of equal length, the text sequences need to be padded or truncated. Padding involves adding a special marker ([PAD]) to the end of the text sequence to make its length reach a specified maximum value; truncating involves cutting the sequence to the specified maximum length. Then, the tokens for each corpus are saved to a text file named tokens.txt, where each token is separated by a space.

[0113] 2. Word frequency statistics:

[0114] First, the `tokens.txt` file is read, and the tokens for each line (one sentence of text corresponds to one line of tokens) are stored in a raw word list. Next, each list of tokens (raw word lists) is traversed, and the frequency of each token in the entire corpus is counted. Then, the total number of tokens in the corpus is calculated, the frequency of each token is normalized, and the probability distribution of each token is calculated. Finally, the probability distribution of the tokens is saved in the text file `frequency.txt`.

[0115] 3. Adaptive mask generation:

[0116] Masks are used to adaptively generate training corpora to ensure that each word in the training corpus has the same probability of being replaced by the mask. In domain-specific corpora, there are often many domain-related high-frequency words. For example, the distribution of some words in threat intelligence in the cybersecurity field is as follows... Figure 2As shown (all distributions have been normalized), the word distribution is extremely uneven.

[0117] If we use existing static masking strategies to randomly mask words in the corpus, for example, each token is selected for masking with a 15% probability (e.g., ... Figure 3 As shown, during the masking operation, the token is replaced with the [MASK] flag with an 80% probability, remains unchanged with a 10% probability, and is randomly replaced with another token from the BERT dictionary vocab.txt with a 10% probability. Therefore, the distribution of the lexical units subjected to the masking operation is as follows: Figure 4 As shown, the uneven distribution of tokens leads to uneven distribution of masks. High-frequency tokens are replaced with masks with a high probability. Therefore, the number of low-frequency words replaced with masks in the final training corpus is small, and the BERT model has difficulty learning the semantic representation of low-frequency words.

[0118] Compared with existing technologies, the embodiments of this application improve the masking strategy of the original BERT. The improved strategy is divided into two stages. First, the frequency.txt file is read to obtain the frequency distribution of tokens. Simultaneously, each line of tokens in the tokens.txt file is read as a tokens list (the original text corpus is split into several text corpus segments based on punctuation marks, and each text corpus segment corresponds to an original word list).

[0119] The process of generating a mask for each token in the tokens list consists of two stages: The first stage involves iterating through each token in the tokens list, from... Frequency of obtaining this token p token and the length of the tokens list Then the token is used The probability magnitude (pre-sampling probability) is added to the pre-sampling term list. In order to ensure that low-frequency tokens are added to the pre-sampled token list with a high probability;

[0120] The second stage involves calculating the pre-sampled lexicon list. length , Each token in The probability is added to the final mask list. In this process, 15% of the tokens in each training statement are replaced with masks, and the tokens corresponding to the mask positions are saved to the word tag list. .in, The calculation method is as follows:

[0121]

[0122] During the masking operation, the mask list is traversed. ,for Each word in the list The tokens list The corresponding token is replaced with the [MASK] flag with an 80% probability, retains its original token with a 10% probability, and is randomly replaced with another token from the BERT dictionary vocab.txt with a 10% probability. The mask distribution generated using the adaptive masking strategy is as follows: Figure 5 As shown.

[0123] 4. Model training:

[0124] The list of tokens (masked words) after masking a single corpus is input into a pre-trained BERT model. The BERT model outputs the predicted token probability at the mask position. Secondly, through the analysis of... Perform OneHot encoding and calculate the cross-entropy loss between the real token label and the predicted label, as shown in the following formula:

[0125]

[0126] in, This represents the number of tokens that were replaced with masks in a single corpus. and yes dimensional vector; This represents the number of words in the BERT model's dictionary, vocab.txt. Through multiple iterations until the cross-entropy loss no longer decreases, a fine-tuned BERT model for a specific domain is obtained.

[0127] It should be noted that when fine-tuning a general GLM or other general models based on masking strategies using domain-specific corpora, an adaptive masking strategy can be used during the mask generation process to ensure that all words are uniformly replaced with masks, thereby better learning the semantic representation of each word.

[0128] Compared with the prior art, the embodiments of this application have the following beneficial effects:

[0129] 1. When fine-tuning BERT using domain-specific corpora, the frequency distribution of each token in the corpus is calculated. This ensures that in the first stage of the mask generation strategy, low-frequency tokens are replaced with masks with a higher probability. In the second stage, the probability of each token being replaced with a mask is adaptively calculated based on the number of pre-sampled tokens and the total number of tokens in a single corpus. This ensures that tokens of different frequencies have an equal chance of being replaced with masks, while guaranteeing that 15% of the tokens in each corpus are replaced with masks, thus preserving most of the domain-specific contextual information.

[0130] 2. By adaptively and dynamically generating masks, each token is eventually replaced with a mask at the same frequency. This avoids the situation where high-frequency tokens are frequently replaced with masks, leading to overtraining, while low-frequency words are undertrained. This ensures the generation of diverse masks, enabling the BERT model to learn complex semantic representations during fine-tuning.

[0131] Please refer to Figure 6 , Figure 6 The diagram illustrates a block diagram of a fine-tuning device for a BERT model provided in some embodiments of this application. It should be understood that this fine-tuning device for the BERT model is similar to that described above. Figure 1 Corresponding to the method embodiments, it is able to execute the various steps involved in the above method embodiments. The specific functions of the BERT model fine-tuning device can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here.

[0132] Figure 6 The BERT model fine-tuning device includes at least one software functional module that can be stored in memory or embedded in the BERT model fine-tuning device in the form of software or firmware. The BERT model fine-tuning device includes:

[0133] The word frequency statistics module 610 is used to generate an original word list based on the text corpus and obtain word frequency statistics information of the original word list;

[0134] The masking module 620 is used to determine the masking generation strategy based on word frequency statistics, and to perform masking processing on the original word list according to the masking generation strategy to obtain the masking processing result.

[0135] The model fine-tuning module 630 is used to iteratively learn the pre-trained BERT model based on the masking processing results until the preset conditions are met, thus obtaining the fine-tuned BERT model.

[0136] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention. The BERT model fine-tuning device provided by the embodiments of the present invention can implement the BERT model fine-tuning method provided by any one of the method embodiments of the present invention.

[0137] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0138] like Figure 7 As shown, some embodiments of this application provide an electronic device 700, which includes: a memory 710, a processor 720, and a computer program stored on the memory 710 and executable on the processor 720. When the processor 720 reads the program from the memory 710 via a bus 730 and executes the program, it can implement any of the methods included in the above-described BERT model fine-tuning method.

[0139] Processor 720 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 720 can be a microprocessor.

[0140] The memory 710 can be used to store instructions executed by the processor 720 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 720 of this disclosure embodiment can be used to execute the instructions in the memory 710 to implement the methods shown above. The memory 710 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0141] Some embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, describes the method described in the method embodiments.

[0142] Some embodiments of this application also provide a computer program product that, when run on a computer, causes the computer to perform the methods described in the method embodiments.

[0143] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0144] It should be understood, in the several embodiments provided in this application, that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0145] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0146] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0147] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0148] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0149] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for fine-tuning a BERT model, characterized in that, include: Generate an original word list based on the text corpus, and obtain the word frequency statistics of the original word list; Based on the word frequency statistics, a mask generation strategy is determined, and the original word list is masked according to the mask generation strategy to obtain the masking result. Based on the masking results, the pre-trained BERT model is iteratively learned until the preset conditions are met, resulting in a fine-tuned BERT model. The process of determining a masking strategy based on the word frequency statistics and masking the original word list according to the masking strategy to obtain a masking result includes: Based on the word frequency statistics, the original word list is pre-sampled to obtain a pre-sampled word list. The final generation probability is determined based on the preset mask generation probability and the comparison of the number of words in the original word list and the pre-sampled word list. Based on the pre-sampled word list and the final generation probability, the original word list is masked to obtain the masking result. The step of determining the final generation probability based on the preset mask generation probability and the comparison of the number of words in the original word list and the pre-sampled word list includes: Obtain the target ratio of the number of lexical units in the original lexical unit list to the number of lexical units in the presampled lexical unit list; The final generation probability is determined based on the product of the preset mask generation probability and the target ratio; Alternatively, the step of determining a masking strategy based on the word frequency statistics and masking the original word list according to the masking strategy to obtain a masking result includes: The frequency of occurrence of each word in the original word list is obtained based on the word frequency statistics information; The probability of each word being extracted from the original word list is determined based on the frequency of occurrence of each word. The original word list is masked based on the extraction probability of each word to obtain the masking result.

2. The fine-tuning method for the BERT model according to claim 1, characterized in that, The step of pre-sampling the original word list based on the word frequency statistics to obtain a pre-sampled word list includes: The frequency of occurrence of each word in the original word list is obtained based on the word frequency statistics information; The probability of each word being extracted from the original word list is determined based on the frequency of occurrence of each word. Based on the probability of each word being extracted, the original word list is pre-sampled to obtain a pre-sampled word list.

3. The fine-tuning method for the BERT model according to claim 1 or 2, characterized in that, The sum of the probability of each word being extracted and the frequency of occurrence of the corresponding word is 1.

4. The fine-tuning method for the BERT model according to claim 1, characterized in that, The step of masking the original word list according to the masking generation strategy to obtain the masking result includes: A list of terms to be processed is determined according to the mask generation strategy; wherein, the list of terms to be processed includes several mask terms that are determined to be masked and processed; The mask position is determined according to the list of words to be processed, and the target words in the original word list corresponding to the mask position are masked to obtain the mask word list. The target words corresponding to the mask position are saved to the word tag list. The word tag list and the mask word list are used as the masking processing result.

5. The fine-tuning method for the BERT model according to claim 4, characterized in that, The step of iteratively learning the pre-trained BERT model based on the mask processing result until a preset condition is met to obtain a fine-tuned BERT model includes: Based on the pre-trained BERT model, the prediction results corresponding to the masked word list are obtained, and based on the word label list, the true word label results are determined. Obtain the cross-entropy loss between the predicted result and the actual word tag result; The pre-trained BERT model is iteratively trained until the cross-entropy loss no longer decreases, resulting in a fine-tuned BERT model.

6. The fine-tuning method for the BERT model according to claim 1, characterized in that, The step of generating an original word list based on the text corpus and obtaining word frequency statistics of the original word list includes: The acquired text corpus is cleaned, and the cleaned text corpus is then divided into several text corpus segments; Each text corpus segment is divided into lexical units based on preset segmentation rules, resulting in an original list of lexical units corresponding to each text corpus segment.

7. A fine-tuning device for a BERT model, characterized in that, include: The word frequency statistics module is used to generate an original word list based on the text corpus and obtain word frequency statistics information of the original word list; The masking module is used to determine a masking generation strategy based on the word frequency statistics information, and to perform masking processing on the original word list according to the masking generation strategy to obtain the masking processing result. The model fine-tuning module is used to iteratively learn the pre-trained BERT model based on the mask processing results until the preset conditions are met, thereby obtaining a fine-tuned BERT model. The mask processing module is specifically used for: Based on the word frequency statistics, the original word list is pre-sampled to obtain a pre-sampled word list. The final generation probability is determined based on the preset mask generation probability and the comparison of the number of words in the original word list and the pre-sampled word list. Based on the pre-sampled word list and the final generation probability, the original word list is masked to obtain the masking result. The step of determining the final generation probability based on the preset mask generation probability and the comparison of the number of words in the original word list and the pre-sampled word list includes: Obtain the target ratio of the number of lexical units in the original lexical unit list to the number of lexical units in the presampled lexical unit list; The final generation probability is determined based on the product of the preset mask generation probability and the target ratio; Alternatively, the mask processing module is specifically used for: The frequency of occurrence of each word in the original word list is obtained based on the word frequency statistics information; The probability of each word being extracted from the original word list is determined based on the frequency of occurrence of each word. The original word list is masked based on the extraction probability of each word to obtain the masking result.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the fine-tuning method of the BERT model according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the fine-tuning method for the BERT model as described in any one of claims 1-6.

Images