Two-stage training method for lightweight Chinese-English bilingual large language model

Abstract

The invention provides a two-stage training method for a lightweight Chinese-English bilingual large language model. The method comprises the following steps: step 1, collecting and preprocessing Chinese-English training data; the preprocessing process comprises special mark clearing, sensitive data clearing, data duplicate removal, advertisement removal and formatting processing; 2, performing first-stage high-resource language training, namely performing model training by using English; 3, second-stage bilingual mixed training is carried out, and the Chinese data and the English data are mixed for second-stage training by adopting language migration training; wherein the step 3 comprises the following steps: step 31, performing word list expansion; and step 32, language migration training. The bilingual processing capability can be improved.

Description

Technical Field

[0001] The present application relates to the fields of natural language processing and information technology, and in particular to a two-stage training method for a lightweight Chinese-English bilingual large language model. Background Art

[0002] With the rapid development of computer technology and information technology, large language models have made significant progress in the field of natural language processing. The parameter size of these models has grown from millions to billions or even trillions, which has greatly improved their performance in various natural language processing tasks, enabling them to perform diverse tasks from human-computer dialogue to mathematical problem solving. For example, large models such as ChatGPT, GPT-4, PaLM-2, LLaMA, Falcon, and Baichuan-2 have shown great potential in general artificial intelligence and have aroused widespread public interest.

[0003] However, the application of these models in specific domains does not usually require extensive general knowledge or domain-independent capabilities, such as program execution. Instead, models should focus on well-defined tasks and domain-specific knowledge or capabilities. Although large models demonstrate broad knowledge coverage and diverse capabilities, they do not perform as well as specialized lightweight models on specific skills or verticals. For example, Phi-1 is a model with only 1.3 billion parameters, which significantly outperforms larger models in this area after being trained on high-quality data for code generation. In addition, due to resource constraints, developers in the field are often unable to effectively apply large language models to specific domains.

[0004] The above situation shows that although large language models have made outstanding achievements in handling various natural language processing tasks, they still face many challenges in handling tasks in specific fields. First, the large parameter scale of large models leads to insufficient flexibility in specific fields, and also increases the cost of deployment and maintenance. Secondly, these models are not as adaptable and efficient as lightweight models optimized for specific tasks in specific fields. In addition, although large models are highly versatile, they still lack depth and accuracy in certain fields. Therefore, developing a language model that can maintain the broad knowledge coverage of large models and be optimized for specific tasks and fields has become a major challenge in the industry.

[0005] Another key technical challenge is the acquisition of large amounts of high-quality text data required to train large-scale language models. This challenge is particularly significant for Chinese models, as high-quality public Chinese data resources are relatively limited. Summary of the invention

[0006] This application provides a two-stage training method for a lightweight Chinese-English bilingual language model, which can be used to solve the technical problem of large amounts of high-quality text data required for training large-scale language models.

[0007] The present application provides a two-stage training method for a lightweight Chinese-English bilingual large language model, the method comprising:

[0008] Step 1: Collect and preprocess Chinese and English training data; the preprocessing process includes special tag removal, sensitive data cleaning, data deduplication, advertisement removal, and formatting;

[0009] Step 2: The first phase of high-resource language training, i.e., model training using English;

[0010] Step 3: Conduct the second phase of bilingual mixed training, using language transfer training to mix Chinese data and English data for the second phase of training;

[0011] Wherein, step 3 comprises:

[0012] Step 31, expand the vocabulary;

[0013] Step 32: language transfer training.

[0014] Furthermore, the formatting process includes:

[0015] Delete extra spaces, tabs, paragraph breaks and other invisible symbols before and after each paragraph in the text content;

[0016] Special space characters are replaced with standard spaces; '\n' is selected as the unified line terminator to replace all original line terminators in the text; to avoid multiple consecutive spaces in the text caused by the replacement operation, consecutive spaces are merged into a single space through regular expressions; unnecessary spaces between Chinese characters are deleted using regular expressions; Traditional Chinese characters are retained.

[0017] Further, remove advertisements, including:

[0018] Using repeated advertisement text blocks detected under different domain names as training data, the Latent Dirichlet Allocation (LDA) algorithm is used to construct an advertisement phrase vocabulary.

[0019] First, the model parameters, including the topic distribution of each document and the word distribution of each topic, are randomly initialized; for each document in the model, each word is initially randomly assigned to a topic; during the iteration process, the initial assignment is continuously updated according to the probability calculated by the model to reflect more accurate topic attribution; the model calculates and reallocates each word to the topic that maximizes its probability of appearing in the current document, which involves combining the probability of the current word under the current topic and the proportion of the current topic in the entire document;

[0020] Subsequently, based on the new assignment, the topic distribution of the document and the word distribution of the topic are updated. This process is repeated, and the topic assignment and update parameters are adjusted each time based on the previous results until the topic assignment is stable or the preset number of iterations is reached. After training the LDA model, the model outputs the word distribution of each topic;

[0021] Check the topics directly related to the ad content and select the words with high weights; make a list of the words with high weights as the key ad words list;

[0022] The keyword advertising word list will be used to identify advertisements in text content. When the appearance rate of advertising phrases in the text content exceeds 50%, the content will be re-filtered.

[0023] Furthermore, step 2, the first stage of high-resource language training, i.e., model training using English, includes:

[0024] Step 21, perform text segmentation, and use the byte pair encoding segmentation algorithm to build an English word segmenter and word list.

[0025] Step 22, performing text encoding;

[0026] Step 23, alternately use full attention and local attention mechanisms to set the attention layer;

[0027] Step 24, setting the residual network, the normalization layer and the fully connected network;

[0028] Step 25, set the output layer and perform autoregressive training;

[0029] Step 26, perform the first stage of training;

[0030] Step 24, setting the residual network, the normalization layer and the fully connected network;

[0031] After each layer of attention mechanism, a residual network and layer normalization operation are used to normalize the output vector to improve the stability of the model, which can be described as a matrix operation:

[0032] H l =LayerNorm(H l-1 +Attn(O l-1 ))

[0033] Among them, H 1 is the hidden variable representation of the lth layer, O l-1 is the hidden variable output of the l-1th layer, Attn(·) is the attention network of the corresponding layer, and LayerNorm is the layer normalization operator; after the residual network, a fully connected network is used to model language knowledge, which can be described as a matrix operation:

[0034] O l =LayerNorm(H l +GeLU(WHl))

[0035] Where W is the trainable weight parameter and GeLU is the nonlinear activation function;

[0036] Step 25, set the output layer and perform autoregressive training;

[0037] The model output layer is based on the representation O of the last layer. L , which is used to predict the conditional probability of the word at each position in the text sequence, is expressed as the following formula:

[0038]

[0039] Among them, W e ∈R |V|×d Unsigned word vector matrix, |V| is the word table size, is the representation of the i-th word by the last encoding layer;

[0040] By maximizing the likelihood estimate of the text sequence, the model is trained to accurately predict the current word based on the given historical sequence, optimizing the following loss function:

[0041]

[0042] Where θ represents the model parameters and n represents the sequence length.

[0043] Furthermore, step 21, text segmentation is performed, and an English word segmenter and word list are constructed using a byte pair encoding word segmentation algorithm, including:

[0044] First, the vocabulary is initialized to the set of all characters in the text. Each character is regarded as a separate word or subword unit. In the given text data, the frequency of occurrence of all adjacent character pairs, i.e., byte pairs, is counted.

[0045] The most frequently occurring character pairs are selected and merged into a new unit, which is then added to the vocabulary. The counting and merging steps are repeated, merging the most frequently occurring byte pairs each iteration until the preset vocabulary size is reached.

[0046] After the vocabulary is constructed, the data is tokenized using the longest match algorithm; starting from the beginning of the text, an attempt is made to match the longest string that is a valid entry in the vocabulary; if a match is found, the current string is identified as a word unit, and the algorithm jumps to the next character after the current string and repeats the process; if a longer match cannot be found at some point, the algorithm backs up and tries a shorter string until a valid vocabulary match is found; after tokenization, the data is converted into a sequence of word units;

[0047] The input is constructed using a chunking strategy, by concatenating all data from different samples and cutting them into a fixed maximum length as training input;

[0048] After word segmentation and chunking, the collected data can be processed into training data with uniform word length for the next step of the training process.

[0049] Furthermore, step 22, performing text encoding, includes:

[0050] For a word sequence of uniform length w=w1w2...w n , the input layer converts the word unit into a dense vector; each word unit w i is mapped into a vector form e i , used to represent the semantics of a word; position information is introduced in the form of position encoding; for each position i, a learnable position vector p is set i Finally, the word w i The input vector is:

[0051] v i =e i +p i .

[0052] Further, step 23, alternately using full attention and local attention mechanisms to set attention layers, includes:

[0053] On different Transformer layers, the model alternates between full attention and local attention. In odd-numbered layers, the model uses full attention, which enables each word to be correlated with all other words in the entire sequence. In even-numbered layers, the model uses local window attention.

[0054] For odd-numbered layers, global attention is used and the matrix operation is expressed as:

[0055] Q=W Q X, K = W K X, V = W K X

[0056]

[0057] Among them, X represents the input matrix including the output of the previous layer, or the input word vector sequence, W Q , W K , W V A trainable weight matrix used to calculate the query, key, and value. d is the dimension of the model. M is the autoregressive mask matrix, which is a matrix with negative infinity in the upper triangle and 0 in the lower triangle.

[0058] For even layers, local attention is used, and the matrix operation is expressed as:

[0059]

[0060] Among them, w is the local window size, which means that only the relevant information within the window size is concerned.

[0061] Further, step 26, performing the first stage training, including:

[0062] The model training process adopts the gradient descent method and uses the Adam optimizer for back propagation;

[0063] Determine the data ratio for training in the first stage, as shown in Table 2; all types of data are evenly mixed and randomly sampled without replacement;

[0064] Table 2: Proportion and quantity of English data trained in the first stage

[0065]

[0066] The data ratio of each data training in the first stage is consistent with Table 2.

[0067] Furthermore, step 31, expanding the vocabulary, includes:

[0068] Based on the collected Chinese data, the Chinese data is processed using a byte pair encoding word segmentation algorithm to generate a Chinese word list; the newly generated Chinese word list is merged with the English word list;

[0069] Compare each word in the newly generated Chinese vocabulary with the word in the original English vocabulary in turn. If the word does not appear in the original English vocabulary, add the word to the end of the original vocabulary; if the word already appears in the original English vocabulary, skip it;

[0070] After expanding the vocabulary, the word vector matrix E is expanded. Since the vector corresponding to the original word unit has been trained, a new word vector E' is randomly initialized for the newly expanded word unit, and the new word vector is spliced ​​after the original word vector matrix.

[0071] Furthermore, step 32, language transfer training, includes:

[0072] The data sampling was conducted with the same language ratio, i.e. the total number of Chinese and English words in the second phase of training was the same;

[0073] Based on the amount of data collected and the impact of each type of data on the model performance, the data ratios shown in Table 4 were determined;

[0074] For English data, all data were trained in the first stage, so the full amount of data was still used for training in the second stage; other types of data were never sampled from the data used in the first stage; during the training process, all types of data were evenly mixed and randomly sampled without replacement;

[0075] To ensure that each batch is input in a cross-pattern of Chinese and English data during model training, one sample is taken from English data and one from Chinese data until the training batch size is reached and then input into the model for training;

[0076] Table 4: Data ratio and quantity table for training in each stage

[0077]

[0078]

[0079] During the second stage of training, the data ratio and quantity follow Table 4.

[0080] This application has the following effects:

[0081] Technical Effect 1: Resource Optimization and Rapid Deployment

[0082] The method proposed in the present invention can train a lightweight bilingual large language model, which greatly reduces the number of parameters and reduces resource consumption during fine-tuning. The model is not only faster in operation speed, but also significantly reduces the deployment and operation costs in cloud services and edge computing environments. Therefore, the model provided by the present invention is more suitable for rapid deployment and maintenance, especially in resource-constrained environments such as mobile devices and embedded systems. In addition, the high efficiency and low resource requirements of the model also make its application in specific fields such as finance, medical or legal industries more practical and feasible.

[0083] Technical Effect 2: Efficient bilingual processing capabilities

[0084] Through the innovative two-stage pre-training method, the model trained by this patented method effectively improves the bilingual processing capabilities of Chinese and English while maintaining a small scale, so that the model can not only process mixed texts in Chinese and English, but also effectively transfer and apply knowledge between different languages, enhancing the applicability of the model in a multilingual environment. In addition, this method solves the data collection and quality issues of low-resource languages ​​such as Chinese, and improves the performance of the model in Chinese, especially when processing professional terms and domain-specific languages.

[0085] Technical Effect 3: Improved Flexibility and Adaptability

[0086] The two-stage training method in this patent is highly flexible and adaptable. The model trained based on this method can be effectively adjusted according to different language and field requirements, thereby achieving more accurate and efficient language processing in multilingual and multi-field environments. This flexibility enables the model to easily adapt to different application scenarios, such as cross-cultural communication, international business, and multilingual information processing, greatly improving the breadth and effectiveness of its practical application.

[0087] In general, this patented technical solution not only solves the optimization problem of large-scale language model applications in specific fields, but also provides an innovative and practical solution for efficient language processing in a multilingual environment. These technical effects jointly promote the development and widespread application of language model technology in practical industrial applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 A flow chart of the method provided in the embodiment of the present application;

[0089] Figure 2 A data processing flow chart provided for an embodiment of the present application;

[0090] Figure 3 A model structure diagram provided for an embodiment of the present application;

[0091] Figure 4 This is an example dialog box diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0092] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0093] The lightweight Chinese-English bilingual model trained by the method provided in this application maintains high language processing performance while significantly reducing the number of parameters and resource consumption during fine-tuning. This lightweight model is not only more suitable for rapid deployment and maintenance, but also shows higher efficiency and practicality in applications in specific fields.

[0094] The present invention adopts an innovative two-stage training method. In the first stage, the present invention uses widely available high-quality English data for basic training of the model. The training in this stage aims to build the basic language understanding and processing capabilities of the model. In the second stage, the present invention selects a small amount of high-quality Chinese data for alignment transfer training. This method not only solves the problem of obtaining Chinese data, but also through this bilingual training strategy, the model can effectively improve its bilingual processing capabilities for Chinese and English while maintaining a small scale. In addition, this two-stage training method also has higher flexibility and adaptability, and can be effectively adjusted according to the needs of different languages ​​and fields, thereby achieving more accurate and efficient language processing in a multilingual environment.

[0095] The main technical steps for solving the problem of the present invention are:

[0096] Step 1, collect text data in the Internet environment, including encyclopedias, books, conversations, papers, web page texts, etc., propose a heuristic quality filtering method to filter the data quality, and finally use the local hash algorithm to deduplicate the collected data to improve the diversity of the data.

[0097] Step 2, the first stage of training, uses public high-quality English data to start training the model. The model will acquire simple language skills and basic knowledge in this stage.

[0098] Step 3, the second phase of training, uses the collected and quality-filtered Chinese data to perform language transfer training on the model. In this phase, in order to ensure the retention of the knowledge learned by the model in the first phase, data with the same distribution as the training data in the first phase are selected for training at the same time to align the language ability and knowledge of Chinese and English.

[0099] The specific contents are elaborated in detail below.

[0100] Step 1: Collect and process Chinese and English training data.

[0101] The present invention uses a large amount of high-quality English data as the language ability and knowledge for model construction, and then uses bilingual data for language migration. Therefore, it is necessary to collect and construct bilingual training data in Chinese and English. For English data, this application uses the Pile dataset, an 800GB dataset containing 22 diverse sources such as web crawling, academic resources, and books. In addition, in order to improve the basic capabilities and factual reliability of the model, the weight of the corresponding content in the training process is increased by upsampling.

[0102] For Chinese pre-training data, data from the WuDao project and CBook, as well as open source materials from multiple Chinese websites, are integrated. The WuDao project covers 3 billion web pages with high text density, and these contents are deeply cleaned to improve data quality and filter out privacy-related information. Only 200GB of WuDao is available in an open version, and this part of the data covers multiple fields such as Wikipedia, news, and comments. Another key data source is CBook, a large-scale Chinese book corpus collected by the Natural Language Processing Laboratory of Fudan University. CBook contains the content of more than 100,000 books, and the texts of these books are extracted from multiple formats (such as PDF, MOBI, and EPUB). Since the above data sets total about 300GB, they are not enough to train large-scale language models. Due to the rapid development of the Internet environment in recent years, web page texts usually provide sufficient data. The present invention also collects a large amount of Chinese text from various websites. To ensure the consistency, diversity, and quality of these self-collected data. This patent carries out a strict preprocessing process, including special tag removal, sensitive data cleaning, data deduplication, and comprehensive screening to remove various low-quality content, such as advertisements and inappropriate website materials.

[0103] Use the web page parser Beautiful Soup to accurately extract and clean the web page text content, eliminate useless HTML, CSS and JS identifiers, emoticons and other abnormal symbols in the web page, and ensure the fluency and usability of the text content. Considering that the purpose is to train a single-modal plain text language model, we focus on extracting plain text data from the web page, traversing and analyzing all text-related nodes of the web page, including the title ( <h1> to< / h1> <h6>),paragraph( ),span( ),piece( ) and lists ( ), using the powerful parsing ability of Beautiful Soup to extract the text content. In order to ensure the purity of the extraction results, images are ignored during the extraction process ( ) and Video( <video>) and other non-text elements.

[0104] Since web page content may come from a variety of different websites, the character encoding of each site may be different, so character encoding detection and processing are performed; the chardet library is used to identify the encoding of the extracted text in order to accurately understand and process the data. The specific encoding will be identified, and in order to facilitate subsequent data processing and analysis, the original data will be converted to UTF-8 encoding; during the transcoding process, if a byte sequence that cannot be recognized or cannot be correctly converted is encountered, including special or rare character encodings, mixed encodings of text, or non-text content, the unconverted content will be discarded to maintain the high quality and consistency of the data set.

[0105] In order to unify the format of the text crawled from the web page, formatting is performed. Delete the extra spaces, tabs, paragraph breaks and other invisible symbols before and after each paragraph in the text content. Special space characters, including Unicode U+00A0 and U+200B, are replaced with standard spaces. In view of the fact that the line terminators between different operating systems may be inconsistent, '\n' is selected as a unified line terminator to replace all the original line terminators in the text. For other invisible characters, such as carriage return (U+000D), soft line break (U+00AD) and other control characters, they are also replaced uniformly. In order to avoid the replacement operation causing multiple consecutive spaces in the text, the consecutive spaces are merged into a single space through a regular expression. Since these web page data are only for Chinese web pages, and spaces between characters should not usually appear in Chinese text, regular expressions are used to delete unnecessary spaces between Chinese characters. It should be noted that this application retains traditional Chinese characters, so that the model can obtain learning materials from classical literature or poetry.

[0106] The preliminary quality assessment of web page content is carried out by calculating the text density. The text density is defined as the ratio of the number of valid text characters in a web page, including Chinese characters, letters, and numbers, to the total number of characters in the web page (including HTML tags, JavaScript code, CSS styles, etc.). In order to accurately determine which text density threshold can effectively filter out high-quality data, an assessment is conducted. Specifically, 1000 web page samples are randomly selected from the collected data, and the samples are evaluated to determine whether each web page contains sufficient information and mark the quality. Calculate the text density of the samples, and iterate through different thresholds from 10% to 90%, calculate the F1 score for classifying the web pages at each threshold, and use this to evaluate the accuracy of the classification effect. Through this process, a text density threshold of 75% can most effectively judge the data quality of the web page, and 75% is used as the text density threshold. During the evaluation and threshold search process, special types of web pages, such as comic web pages containing pictures and their short descriptions, or minimalist web pages containing only titles, although these web pages have short content but may still be information-rich. Therefore, using only text density as a filtering criterion is not always effective in these cases. Considering these factors, a method that combines text density and the number of valid characters is used for preliminary quality filtering. Finally, it is determined that any web page with a text density lower than 75% or containing less than 100 valid characters will be ignored to ensure that the high-quality standards of the dataset are maintained.

[0107] Since the WuDao project also extracts content from web pages, and some websites will repost the same information, in order to reduce data duplication and improve data quality, duplicate removal is performed on the data samples. In the data preprocessing stage, first, a specialized word segmentation tool is used to segment the collected web page content. Considering that this dataset mainly processes Chinese, the jieba word segmentation tool is selected, which can effectively identify the word boundaries in Chinese. After word segmentation, in order to reduce the impact of stop words (such as frequently appearing words like "de", "he", etc. that contribute little to understanding the text theme) on the text similarity assessment, multiple stop word lists are combined, such as the Harbin Institute of Technology stop word list and the Baidu stop word list, and stop words are removed from the word segmentation results.

[0108] The TF-IDF method is applied to evaluate the importance of each word in the text. First, calculate the term frequency (TF), which is the number of times a word appears in a specific document divided by the total number of words in the document; then calculate the inverse document frequency (IDF), which is the logarithm reciprocal of the total number of documents based on the number of documents containing the word. Multiply the TF value of a word by its IDF value, and the higher the resulting TF-IDF value, the more important the current word is in the document. The SimHash algorithm is used to evaluate and compare text similarity, and the steps include:

[0109] Perform an MD5 hash operation on each word in the text to generate a 64-bit bit string;

[0110] Then, the hash value of each word is weighted according to its TF-IDF value, and the weighted hash values ​​of all words are accumulated.

[0111] On the accumulated result, each bit is judged. If the weighted sum is greater than zero, the corresponding bit is set to 1 in the SimHash value, otherwise it is set to 0. Each document is converted into a SimHash value of fixed length.

[0112] The Hamming distance is used to measure the difference between two SimHash values, that is, to compare the number of different characters in the same position of two SimHash values; if the Hamming distance of the SimHash values ​​of two documents is less than or equal to 3, the two documents are highly similar in content and are marked as duplicate samples. Only the one with the longest number of characters among the similar samples is retained to optimize the content quality and uniqueness of the data set. This series of sophisticated processing steps ensures the accuracy and availability of the data, providing a high-quality foundation for subsequent data analysis and model training.

[0113] Web pages often contain personal privacy and sensitive information. Sensitive vocabulary and heuristic methods are used to detect and filter information. First, this patent collects filtering vocabulary from different sources, which are maintained by government agencies, social organizations or online communities. Then, consult relevant laws and regulations to extract words that are explicitly prohibited, including personal privacy and sensitive information. In order to identify and process this information, a heuristic regular expression method is used for detection, including: 1. ID card number: ID card number usually has a specific format, usually 18 digits (or 15 digits in the old version), of which the first 17 digits are numbers, and the last digit may be a number or the letter 'X'. The regular expression is: \b\d{15}|\d{17}[0-9Xx]\b; 2. Phone number: The format of the phone number may vary by country or region. Taking Chinese mobile phone numbers as an example, they are usually 11 digits long and start with 1. The regular expression is: \b1[3-9]\d{9}\b; 3. Email: The format of the email address is usually username@domain name. The regular expression is: \b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b; 4. Bank account information: Bank account information has various formats and generally contains a certain length of digital sequence. Chinese bank accounts are generally 16 or 19 digits long. The regular expression is: \b\d{16}|\d{19}\b; 5. Address information: Address information is relatively complex and involves multiple formats, including specific address information such as province, city, district, and street. Regular expressions are used to identify some common keywords, such as province, city, district, street, etc., and format matching text fragments containing specific address keywords. The regular expression is as follows: (?:province|city|district|county|road|street|number|building|unit|room)\b; once personal privacy and sensitive information is identified, the information is replaced with special characters of the corresponding category, such as replacing the ID number with "[ID number]", the phone number with "[phone number]", the email with "[email]", the bank account information with "[bank account]", and the address information with "[address]" to protect user privacy. This process ensures that all identified sensitive information will not be leaked while retaining the structure and readability of the text.

[0114] Web pages often contain duplicate advertising content, especially on multiple pages under the same domain. In order to effectively filter out duplicate advertising content under the same domain, set up a self-duplicate content filter, paying special attention to the structural blocks of the web page, including paragraphs. ,title <h1>-< / h1> <h6>, and other containers ; Web page blocks often contain repeated advertising statements, which affects data quality. Group web page content according to domain names, and group web page texts with the same domain name;

[0115] Subsequently, the frequency of occurrence of each text block in each web page is analyzed, and statistics are given to see whether the text in each block appears in other pages. When a text block appears repeatedly in 70% of the same set of pages, the corresponding text block is identified as an advertisement or duplicate content; the text block will be marked and excluded from the final data set. In this way, the patent can accurately filter out repeated advertising content. In addition, repeated advertising text blocks detected under different domain names are used as training data, and the Latent Dirichlet Allocation (LDA) algorithm is used to construct an advertising phrase vocabulary. When training the LDA model, the entire process involves initializing the model parameters and gradually refining these parameters through multiple iterations to achieve convergence.

[0116] First, the model parameters, including the topic distribution for each document and the word distribution for each topic, are randomly initialized. For each document in the model, each word is initially randomly assigned to a topic. During the iteration process, the initial assignment is continuously updated based on the probabilities calculated by the model to reflect more accurate topic attribution. The model calculates and reallocates each word to the topic that maximizes its probability of appearing in the current document, which involves combining the probability of the current word under the current topic and the proportion of the current topic in the entire document. Subsequently, based on the new assignment, the topic distribution of the document and the word distribution of the topic are updated, and this update reflects the change in the corresponding topics of each word in all documents. This process is repeated, and the topic assignment and update parameters are adjusted based on the previous results each time until the topic assignment is stable or the preset number of iterations is reached, so as to ensure that the topic distribution of each document and the vocabulary composition of each topic have reached the data-driven optimal state. After training the LDA model, the model outputs the word distribution of each topic, which reflects the most relevant words for each topic. Look at the topics that are directly related to the ad content and select the words with high weights; these high-weight words are usually keywords that appear frequently in ads, such as "discount", "free", "limited time", etc. Finally, the high-weight words are organized into a list as a key advertising word list; the key advertising word list will be used to identify advertisements in text content. When the appearance rate of advertising phrases in the text content exceeds 50%, the content will be re-filtered.

[0117] Finally, the data quality needs to be evaluated. This patent adopts a manual sampling evaluation method to evaluate text quality. The quality of the overall text data is evaluated by extracting some text samples and manually reviewing them. First, this patent adopts random sampling combined with stratified sampling to ensure the accuracy and reliability of the evaluation results. Afterwards, for the sampled text samples, clear evaluation standards and rules are formulated, such as clarity, logic, coherence, etc., and provided to the evaluators for understanding and execution. Finally, based on the results of manual sampling evaluation, problems and deficiencies in the data set can be identified. According to the evaluation results, take corresponding quality improvement measures.

[0118] Figure 2 The data processing process is shown. Finally, the present invention builds a large Chinese text corpus containing about 300GB of data, covering a variety of different fields from novels to law. The collection and processing of these data is intended to provide a comprehensive and diverse data foundation for the training of language models.

[0119] The final data obtained by processing is shown in Table 1:

[0120] Table 1: Data overview

[0121]

[0122] Step 2: The first phase of high-resource language training is to use English for model training.

[0123] With the global popularity of the Internet, a large amount of rich text data is stored online, including open access books, social media content, etc. These data show significant statistical differences in different languages, especially in quantity and quality. In this context, English, as one of the most widely used languages ​​in the world, has its related high-quality data resources collected and opened to the public on a large scale. In contrast, although Chinese is also one of the most widely used languages ​​in the world, there are relatively few high-quality Chinese data resources available for public use. In order to address this gap and reduce the complexity and cost of collecting Chinese text corpus, the present invention uses English for model training in the first stage to lay the foundation for the basic knowledge and language processing capabilities of the language model.

[0124] Step 21, perform text segmentation, and use the byte pair encoding segmentation algorithm to build an English word segmenter and word list.

[0125] Text segmentation is an indispensable step in natural language processing, especially in the training process of large-scale language models. The main purpose of word segmentation is to convert complex text data into smaller units that can be understood by the model. These units are called tokens. This application uses a byte pair encoding word segmentation algorithm to construct an English word segmenter and word list.

[0126] The core of the byte pair encoding word segmentation algorithm is an iterative character merging mechanism. For all collected data, the byte pair encoding word segmentation algorithm first breaks the text into characters, and then gradually merges the most frequently occurring character pairs.

[0127] First, the vocabulary is initialized to the set of all characters in the text. Each character is regarded as a separate word or subword unit. In the given text data, the frequency of occurrence of all adjacent character pairs, i.e., byte pairs, is counted.

[0128] The most frequently occurring character pairs are selected and merged into a new unit, which is then added to the vocabulary; for example, if "es" is the most common character pair, then "es" will be merged into a new unit "es". Repeat the counting and merging steps, merging the most frequent byte pairs each iteration until the preset vocabulary size is reached. In this process, the vocabulary gradually expands from a single character to include more common byte pairs or character sequences. The key to this method is that it does not rely on a predefined vocabulary, but dynamically generates a vocabulary based on actual data. It is a language-independent data-driven word segmentation algorithm.

[0129] After the vocabulary is built, the data is tokenized using the longest match algorithm; starting from the beginning of the text, an attempt is made to match the longest string that is a valid entry in the vocabulary. This means that the algorithm checks the longest possible string starting from the current position and queries whether this string exists in the vocabulary. If a match is found, the current string is confirmed as a word unit, and the algorithm jumps to the next character after the current string and continues the process; if a longer match cannot be found at some point, the algorithm backs up and tries shorter strings until a valid vocabulary match is found; after tokenization, the data is converted into a sequence of word units.

[0130] In order to perform batch processing when training the model, it is necessary to form input data of the same length. However, due to the different lengths of the original input data, previous methods require padding or truncation of each data. In order to improve the input efficiency of the model and reduce the data waste caused by excessive padding, a block strategy is used to form the input, by splicing all the data from different samples together and cutting them into a fixed maximum length (for example, using 2048 tokens) as training input. This cutting method can effectively utilize the input capacity of the model. Secondly, it enables the model to process longer text sequences, thereby better understanding the long-distance dependencies in the text and reducing the degradation of model performance that may be caused by filling a large amount of irrelevant data.

[0131] After word segmentation and chunking, the collected data can be processed into training data with uniform word length for the next step of the training process.

[0132] Step 22, text encoding

[0133] For a word sequence of uniform length w=w1w2...w n , the input layer converts the word unit into a dense vector; each word unit w i is mapped into a vector form e i , used to represent the semantics of a word; since this patent is aimed at the Transformers architecture, the architecture uses an attention mechanism that cannot perceive position offsets, and introduces position information in the form of position encoding; for each position i, a learnable position vector p is set i Finally, the word w i The input vector is:

[0134] v i =e i +p i

[0135] In this model, the input layer is not just a simple data conversion link, but plays a vital role. It ensures that each word not only carries its semantic information (through e i ), and also contains its position information in the sequence (through p i ). These vectorized encoding results v i A comprehensive and detailed representation is provided for each word, so that the subsequent neural network processing can more accurately capture and understand the complexity of the text. In the model trained by this patented method, each step of processing is aimed at improving the accuracy of data processing and the performance of the model, ensuring that the final output not only reflects the true meaning of the text, but also meets the requirements of the neural network for efficient data processing.

[0136] Step 23, alternately use full attention and local attention mechanisms to set the attention layer;

[0137] The model trained by this patented method uses a method that combines local window attention and hierarchical attention, called cross-local global attention, to optimize the processing of long text sequences. This method effectively balances computational efficiency and depth of information capture by alternating between full attention and local attention mechanisms, aiming to improve efficiency and accuracy when processing long text sequences while reducing the computational burden.

[0138] The model trained by this patented method uses a hierarchical attention mechanism. On different Transformer layers, the model alternates between full attention and local attention. In odd-numbered layers (1st, 3rd, 5th, 7th, 9th, 11th, etc. of the model), the model uses a full attention mechanism to enable each word to calculate relevance with all other words in the entire sequence. This global perspective enables the model to capture a wider range of contextual information and understand long-distance dependencies in the text. In even-numbered layers (2nd, 4th, 6th, 8th, 10th, etc. of the model), the model uses local window attention to focus on capturing detailed context around the word. This alternating strategy between layers not only increases the model's sensitivity to different semantic levels, but also balances computational efficiency and model performance.

[0139] Global attention and local window attention mechanisms are two different attention mechanisms. Global attention calculates the attention score of each input position to all positions, and interacts and integrates information; global attention can capture information at all positions, but requires larger computing resources. Under the local window mechanism, the attention of each word unit is not evenly distributed over the entire text sequence, but is concentrated in a local window around it. This means that each word unit only interacts and exchanges information with a few neighboring words. In this way, the model is able to capture local contextual information in the text, similar to the convolution operation in visual processing. The local window attention mechanism is particularly suitable for capturing short-distance dependencies in text while significantly reducing the computational burden. The two attention mechanisms can be expressed as matrix operations.

[0140] For odd-numbered layers, global attention is used and the matrix operation is expressed as:

[0141] Q=W Q X, K = W K X, V = W K X

[0142]

[0143] Among them, X represents the input matrix including the output of the previous layer, or the input word vector sequence, W Q , W K , W V The trainable weight matrix used to calculate the query, key and value. d is the dimension of the model. M is the autoregressive mask matrix, which is a matrix with negative infinity in the upper triangle and 0 in the lower triangle to ensure autoregression.

[0144] For even layers, local attention is used, and the matrix operation is expressed as:

[0145]

[0146] Among them, w is the local window size, which means that only the relevant information within the window size is concerned.

[0147] This sparse attention mechanism, which combines local window attention and hierarchical attention, provides this method with a powerful ability to process long text sequences. It not only optimizes the use of computing resources and reduces the computational burden, but also ensures that the model can capture a variety of language features from local to global.

[0148] Step 24: Set up the residual network, normalization layer, and fully connected network

[0149] After each layer of attention mechanism, a residual network and layer normalization operation are used to normalize the output vector to improve the stability of the model, which can be described as a matrix operation:

[0150] H l =LayerNorm(H l-1 +Attn(O l-1 ))

[0151] Among them, H 1 is the hidden variable representation of the lth layer, O l-1 is the hidden variable output of the l-1th layer, Attn(·) is the attention network of the corresponding layer, and LayerNorm is the layer normalization operator; after the residual network, this paper uses a fully connected network modeling language knowledge, which can be described as a matrix operation:

[0152] O l =LayerNorm(H l +GeLU(WH l ))

[0153] Where W is a trainable weight parameter, GeLU is a nonlinear activation function; a total of 24 layers of neural network are used, the odd 12 layers are global attention, and the even 12 layers are local window attention. Figure 3 shown.

[0154] Step 25: Set the output layer and autoregressive training

[0155] This step is the core of the model learning process. The output layer of the model trained by this patent method is based on the representation of the last layer. L , which is used to predict the conditional probability of the word at each position in the text sequence, is expressed as the following formula:

[0156]

[0157] Among them, W e ∈R |V|×d Unsigned word vector matrix, |V| is the word table size, is the representation of the i-th word by the last encoding layer.

[0158] The model trained in this patent is a one-way language model that inputs a text sequence w in a natural reading order. It adopts the traditional language model objective, which is to maximize the likelihood estimate of the text sequence. The model is trained to accurately predict the current word based on a given historical sequence, optimizing the following loss function:

[0159]

[0160] Where θ represents the model parameters and n represents the sequence length.

[0161] Step 26, perform the first stage of training;

[0162] In order to effectively optimize this loss function, the model training process uses the gradient descent method and uses the Adam optimizer for back propagation. The Adam optimizer is an efficient gradient descent variant that combines the characteristics of momentum and adaptive learning rate to achieve faster convergence and higher training stability. This optimization strategy enables the model to adjust parameters more accurately during the learning process to adapt to complex and changing language patterns, ultimately achieving the ability to deeply understand and generate language.

[0163] Based on the above steps, this patent considers performance and data volume while determining the data ratio for training in the first stage, as shown in Table 2. In the first stage, 159 billion words of English data were sampled in the English data set according to the ratio in Table 2, that is, about 80 million data samples were used to train the model. During the training process, in order to improve the overall learning ability of the model, all types of data were evenly mixed and randomly sampled without replacement.

[0164] Table 2: Proportion and quantity of English data trained in the first stage

[0165]

[0166] After the first stage, the trained model acquired basic English capabilities. Since the training data ratio deeply considered the impact of different types of data on the model, the model of the first stage of this patent surpassed larger-scale models such as MOSS, GPT-J, and MPT in two English large language model benchmark tests MMLU and AGIEval. The test results are shown in Table 3.

[0167] Table 3: Evaluation results of the model trained in the first stage on the English benchmark

[0168]

[0169] Step 3: Conduct the second phase of bilingual mixed training

[0170] After training in step 2, the model trained by the patent method already has strong English language ability and basic knowledge. Furthermore, in order to be used in the Chinese field, this patent adopts language transfer training, mixing Chinese data and English data for the second stage of training. The second stage of training aligns Chinese and English abilities, and uses higher quality English data to assist in learning Chinese abilities.

[0171] Step 31: Expand the vocabulary

[0172] In step 2.1, this patent uses a byte pair encoding word segmentation algorithm to build an efficient English vocabulary, which enables the model to effectively encode English text data. In order to further enhance the language processing capabilities of the model, especially in processing Chinese data, this patent adopts a vocabulary expansion strategy to add a Chinese vocabulary to the model.

[0173] Similar to the method in step 2.1, based on the collected Chinese data, the Chinese data is processed using a byte pair encoding word segmentation algorithm to generate a Chinese word list; the newly generated Chinese word list is merged with the English word list;

[0174] Each word in the newly generated Chinese vocabulary is compared with the word in the original English vocabulary in turn. If the word does not appear in the original English vocabulary, it is added after the original vocabulary; if the word already appears in the original English vocabulary, it is skipped to avoid unnecessary redundancy and potential confusion, ensuring the efficiency and compactness of the vocabulary, and also guaranteeing the accuracy and flexibility of the model when processing mixed Chinese and English texts.

[0175] After expanding the vocabulary, the word vector matrix E is expanded. Since the vector corresponding to the original word unit has been trained, a new word vector E' is randomly initialized for the newly expanded word unit, and the new word vector is spliced ​​after the original word vector matrix.

[0176] After successfully expanding the vocabulary, the model trained by the patented method now has the powerful ability to process bilingual text data in Chinese and English. In this way, the model trained by the patented method successfully expanded its language processing range, not only being able to accurately process and understand English text, but also being able to deal with Chinese data with the same efficiency.

[0177] Step 32: Language transfer training

[0178] A mixed training method of Chinese and English is used. Chinese data is less than English data. In order to ensure a balance between the two, the same language ratio is used for data sampling, that is, the total amount of Chinese and English words in the second stage of training is the same;

[0179] Based on the amount of data collected and the impact of each type of data on the model performance, the data ratios shown in Table 4 were determined;

[0180] For English data, some types of data, such as encyclopedia and mathematics data, were trained in the first stage, so the full amount of data was still used for training in the second stage; while other types of data were never sampled from the data used in the first stage. During the training process, in order to improve the overall learning ability of the model, all types of data (each language) were evenly mixed and randomly sampled without replacement;

[0181] In order to ensure that the model training process does not favor a certain language, each batch is input using a crossover of Chinese and English data, that is, one piece of English data and one piece of Chinese data are crossovered until the training batch size is reached and then input into the model for training;

[0182] Table 4: Data ratio and quantity table for training in each stage

[0183]

[0184] This strategy enables the model to not only fully absorb and retain the knowledge previously learned on English data during training, but also effectively combine it with the newly acquired Chinese language skills and knowledge. Through this carefully designed training process, this patent finally successfully cultivated a large-scale language model capable of processing Chinese and English bilingual data.

[0185] As for English proficiency, adding Chinese training data will not weaken English proficiency. Due to the addition of some English training data and the alignment of Chinese and English, English proficiency is further improved. The model after the second stage of training has improved on the English benchmark evaluation MMLU and AGIEval as shown in Table 5, compared with the first stage, and has surpassed more and larger models.

[0186] Table 5: Evaluation results of the model trained in the second stage on the English benchmark

[0187]

[0188] On the benchmark tests of the Chinese large language model C-Eval and CMMLU, the model trained by the patented method demonstrated its excellent performance. As shown in Table 6, the model of this patent is not only better than other similar models, but even surpasses larger-scale models in some aspects. This achievement marks an important breakthrough in the field of multilingual processing and demonstrates its great potential and value in practical applications. The model of this patent is not only a technological innovation, but also an important step towards a truly global, multilingual intelligent language processing system.

[0189] Table 6: Evaluation results of the model trained in the second stage on the Chinese benchmark

[0190]

[0191]

[0192] Adaptation case of a city's government services: The bilingual large model trained based on this patented method and adaptively trained for a city's government services can provide citizens with government services such as policy interpretation in the form of dialogue, reduce municipal service expenses, and improve municipal service efficiency.

[0193] Figure 4 Shows questions and answers related to a city. The knowledge comes from the "Notice of the Bureau of Industry and Information Technology on the Redemption of Municipal Production and Efficiency Increase Incentive Funds in the First Quarter of 2023" in the training data

[0194] The method provided by the present invention realizes the application examples of large language models in specific fields, solves the problem that large-scale language models are difficult to optimize in specific fields, and proposes a solution for the implementation of large-scale language model applications in specific fields.

[0195] This paper innovatively proposes a two-stage pre-training method, which uses high-quality, diverse, high-resource languages ​​to perform preliminary basic training on the model, and then uses bilingual training to align multilingual knowledge bases and capabilities, solving the low-resource language quality problems and collection difficulties.

[0196] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application. < / h6> < / video> < / h6>

Claims

1. A two-stage training method for a lightweight Chinese-English bilingual large language model, characterized in that: The method comprises: Step 1: Collect and preprocess Chinese and English training data; the preprocessing process includes special tag removal, sensitive data cleaning, data deduplication, advertisement removal, and formatting; Step 2: The first phase of high-resource language training, i.e., model training using English; Step 3: Conduct the second phase of bilingual mixed training, using language transfer training to mix Chinese data and English data for the second phase of training; Wherein, step 3 comprises: Step 31, expand the vocabulary; Step 32: language transfer training.

2. The method according to claim 1, characterized in that: Formatting, including: Delete extra spaces, tabs, paragraph breaks and other invisible symbols before and after each paragraph in the text content; Special space characters are replaced with standard spaces; '\n' is selected as the unified line terminator to replace all original line terminators in the text; to avoid multiple consecutive spaces in the text caused by the replacement operation, consecutive spaces are merged into a single space through regular expressions; unnecessary spaces between Chinese characters are deleted using regular expressions; Traditional Chinese characters are retained.

3. The method according to claim 1, characterized in that Remove ads, including: Using repeated advertisement text blocks detected under different domain names as training data, the Latent Dirichlet Allocation (LDA) algorithm is used to construct an advertisement phrase vocabulary. First, the model parameters, including the topic distribution of each document and the word distribution of each topic, are randomly initialized; for each document in the model, each word is initially randomly assigned to a topic; during the iteration process, the initial assignment is continuously updated according to the probability calculated by the model to reflect more accurate topic attribution; the model calculates and reallocates each word to the topic that maximizes its probability of appearing in the current document, which involves combining the probability of the current word under the current topic and the proportion of the current topic in the entire document; Subsequently, based on the new assignment, the topic distribution of the document and the word distribution of the topic are updated. This process is repeated, and the topic assignment and update parameters are adjusted each time based on the previous results until the topic assignment is stable or the preset number of iterations is reached. After training the LDA model, the model outputs the word distribution of each topic; Check the topics directly related to the ad content and select the words with high weights; make a list of the words with high weights as the key ad words list; The keyword advertising word list will be used to identify advertisements in text content. When the appearance rate of advertising phrases in the text content exceeds 50%, the content will be re-filtered.

4. The method according to claim 1, characterized in that: Step 2: The first phase of high-resource language training, i.e. model training using English, includes: Step 21, perform text segmentation, and use the byte pair encoding segmentation algorithm to build an English word segmenter and word list. Step 22, performing text encoding; Step 23, alternately use full attention and local attention mechanisms to set the attention layer; Step 24, setting the residual network, the normalization layer and the fully connected network; Step 25, set the output layer and perform autoregressive training; Step 26, perform the first stage of training; Step 24, setting the residual network, the normalization layer and the fully connected network; After each layer of attention mechanism, a residual network and layer normalization operation are used to normalize the output vector to improve the stability of the model, which can be described as a matrix operation: H l =LayerNorm(H l-1 +Attn(O l-1 )) Among them, H l is the hidden variable representation of the lth layer, O l-1 is the hidden variable output of the l-1th layer, Attn(·) is the attention network of the corresponding layer, and LayerNorm is the layer normalization operator; after the residual network, a fully connected network is used to model language knowledge, which can be described as a matrix operation: O l =LayerNorm(H l +GeLU(WH l )) Where W is the trainable weight parameter and GeLU is the nonlinear activation function; Step 25, set the output layer and perform autoregressive training; The model output layer is based on the representation O of the last layer. L , which is used to predict the conditional probability of the word at each position in the text sequence, is expressed as the following formula: Among them, W e ∈R |V|×d Unknown word vector matrix, |V| is the vocabulary size, is the representation of the i-th word by the last encoding layer; By maximizing the likelihood estimate of the text sequence, the model is trained to accurately predict the current word based on the given historical sequence, optimizing the following loss function: Where θ represents the model parameters and n represents the sequence length.

5. The method according to claim 1, characterized in that: Step 21, perform text segmentation, and use a byte pair encoding segmentation algorithm to build an English segmenter and a vocabulary, including: First, the vocabulary is initialized to the set of all characters in the text. Each character is regarded as a separate word or subword unit. In the given text data, the frequency of occurrence of all adjacent character pairs, i.e., byte pairs, is counted. The most frequently occurring character pairs are selected and merged into a new unit, which is then added to the vocabulary. The counting and merging steps are repeated, merging the most frequently occurring byte pairs each iteration until the preset vocabulary size is reached. After the vocabulary is constructed, the data is tokenized using the longest match algorithm; starting from the beginning of the text, an attempt is made to match the longest string that is a valid entry in the vocabulary; if a match is found, the current string is identified as a word unit, and the algorithm jumps to the next character after the current string and continues the process; if a longer match cannot be found at some point, the algorithm backs up and tries a shorter string until a valid vocabulary match is found; after tokenization, the data is converted into a sequence of word units; The input is constructed using a chunking strategy, by concatenating all data from different samples and cutting them into a fixed maximum length as training input; After word segmentation and chunking, the collected data can be processed into training data with uniform word length for the next step of the training process.

6. The method according to claim 1, characterized in that Step 22, performing text encoding, includes: For a word sequence of uniform length input to the model w = w1w2…w n , the input layer converts the word unit into a dense vector; each word unit w i is mapped into a vector form e i , used to represent the semantics of a word; position information is introduced in the form of position encoding; for each position i, a learnable position vector p is set i Finally, the word w i The input vector is: v i =e i +p i 。 7. The method according to claim 1, characterized in that Step 23, alternately use the full attention and local attention mechanisms to set the attention layer, including: On different Transformer layers, the model alternates between full attention and local attention. In odd-numbered layers, the model uses full attention, which enables each word to be correlated with all other words in the entire sequence. In even-numbered layers, the model uses local window attention. For odd-numbered layers, global attention is used and the matrix operation is expressed as: Q=W Q X,K=W K X,V=W K X Among them, X represents the input matrix including the output of the previous layer, or the input word vector sequence, W Q ,W K ,W V A trainable weight matrix used to calculate the query, key, and value. d is the dimension of the model. M is the autoregressive mask matrix, which is a matrix with negative infinity in the upper triangle and 0 in the lower triangle. For even layers, local attention is used, and the matrix operation is expressed as: Among them, w is the local window size, which means that only the relevant information within the window size is concerned.

8. The method according to claim 1, characterized in that: Step 26, performing the first stage of training; including: The model training process adopts the gradient descent method and uses the Adam optimizer for back propagation; Determine the data ratio for training in the first stage, as shown in Table 2; all types of data are evenly mixed and randomly sampled without replacement; Table 2: Proportion and quantity of English data trained in the first stage The data ratio of each data training in the first stage is consistent with Table 2.

9. The method according to claim 1, characterized in that: Step 31, expanding the vocabulary; including: Based on the collected Chinese data, the Chinese data is processed using a byte pair encoding word segmentation algorithm to generate a Chinese word list; the newly generated Chinese word list is merged with the English word list; Compare each word in the newly generated Chinese vocabulary with the word in the original English vocabulary in turn. If the word does not appear in the original English vocabulary, add the word to the end of the original vocabulary; if the word already appears in the original English vocabulary, skip it; After expanding the vocabulary, the word vector matrix E is expanded. Since the vector corresponding to the original word unit has been trained, a new word vector E' is randomly initialized for the newly expanded word unit, and the new word vector is spliced ​​after the original word vector matrix.

10. The method according to claim 9, characterized in that Step 32, language transfer training, includes: The data sampling was conducted with the same language ratio, i.e. the total number of Chinese and English words in the second phase of training was the same; Based on the amount of data collected and the impact of each type of data on the model performance, the data ratios shown in Table 4 were determined; For English data, all data were trained in the first stage, so the full amount of data was still used for training in the second stage; other types of data were never sampled from the data used in the first stage; during the training process, all types of data were evenly mixed and randomly sampled without replacement; To ensure that each batch is input in a cross-pattern of Chinese and English data during model training, one sample is taken from English data and one from Chinese data until the training batch size is reached and then input into the model for training; Table 4: Data ratio and quantity table for training in each stage During the second stage of training, the data ratio and quantity follow Table 4.

Images