Lightweight Chinese-English bilingual big language model training method

Abstract

The invention provides a training method of a lightweight Chinese-English bilingual large language model. The method comprises the following steps: step 1, collecting and preprocessing Chinese-English training data; step 2, carrying out pre-training; step 21, performing text word segmentation; step 22, text coding; step 23, rotation position coding: rotation position coding in an attention layer is adopted in an input layer; step 24, setting an attention mechanism, a residual network, a standardization layer, a full-connection network output layer and an autoregressive training mode; step 25, selecting pre-training data; step 26, carrying out pre-training; each batch is input in a Chinese and English data crossing mode, that is, one piece of data is sampled from each of English data and Chinese data in a crossing mode until the size of the training batch is reached, and then the data is input into the model for training. And step 3, instruction fine tuning training. The method is extremely high in flexibility and adaptability.

Description

Technical Field

[0001] The present application relates to the fields of natural language processing and information technology, and in particular to a method for training a lightweight Chinese-English bilingual large language model. Background Art

[0002] In today's information technology and computer science, natural language processing technology has made significant progress thanks to the rapid development of large language models. However, when these large models are applied to specific fields such as law and medicine, their extensive general knowledge or capabilities are not always required. More importantly, the model should focus on specific tasks and domain-related knowledge. Although these large models excel in terms of breadth of knowledge coverage and diverse capabilities, they may not be as effective in specific fields or skills as lightweight models that are tailored for specific tasks. For example, although the Phi-1 model has only 1.3 billion parameters, its performance has surpassed larger models after being trained on professional data in the field of code generation.

[0003] The industry is currently facing an important problem: although large language models perform well in language understanding, their large number of parameters means that they consume huge computing resources when fine-tuning in specific fields. This problem is particularly prominent in specific industries such as finance, medicine, or law, because in these fields, large models often only need to play their reasoning and knowledge memory functions for that field. The large size of the model and its resource-intensiveness during fine-tuning have become key obstacles to its widespread application in the industry. Therefore, a major challenge in the industry is to develop language models that have both the broad knowledge coverage capabilities of large models and can be optimized for specific tasks and fields. Summary of the invention

[0004] The present application provides a lightweight Chinese-English bilingual large language model training method, which can be used to solve the technical problem that the number of parameters of a large language model consumes huge computing resources when fine-tuning in a specific field.

[0005] This application provides a lightweight Chinese-English bilingual large language model training method, the method comprising:

[0006] Step 1: Collect and preprocess Chinese and English training data; preprocessing includes character encoding detection and processing, preliminary quality assessment of web page content, and deduplication of data samples;

[0007] Step 2: Pre-training; including:

[0008] Step 21, perform text segmentation;

[0009] Step 22, text encoding: Faced with the uniform length word sequence w=w1w2…w input to the modeln , adopting the word-gram mapping method, the word-gram sequence is converted into a dense vector format that can be effectively processed by the neural network at the input layer;

[0010] Step 23, rotational position encoding: the rotational position encoding in the attention layer is used in the input layer; rotational position encoding refers to integrating the position information of the word unit into the vector in a rotational transformation manner;

[0011] Step 24, setting the attention mechanism, residual network, normalization layer, fully connected network output layer and autoregressive training method; wherein FlashAttention2 is used as the attention mechanism;

[0012] Step 25, select pre-training data;

[0013] Step 26, pre-training; each batch is input using a cross-pattern of Chinese and English data, that is, one sample is taken from the English data and one from the Chinese data, until the training batch size is reached and then input into the model for training.

[0014] Step 3: Instruct fine-tuning training.

[0015] Furthermore, step 1, character encoding detection and processing, includes:

[0016] Use the web page parser Beautiful Soup to accurately extract and clean the text content of web pages, eliminate useless HTML, CSS and JS identifiers, emoticons and other abnormal symbols in the web pages, extract plain text data from the web pages, traverse and analyze all text-related nodes of the web pages, including titles <h1>to< / h1> <h6>,paragraph ,span ,piece and list , using Beautiful Soup to extract the text content. Images are ignored during the extraction process and Video <video>Non-text elements;

[0017] Detect and process character encodings; use the chardet library to identify the encoding of the extracted text; identify the specific encoding and convert the original data into UTF-8 encoding; during the transcoding process, if you encounter a byte sequence that cannot be recognized or cannot be correctly converted, including special or rare character encodings, mixed encodings of text, or non-text content, then discard the content that cannot be converted.

[0018] Furthermore, the initial quality assessment of web page content includes:

[0019] Perform a preliminary quality assessment of web page content by calculating text density;

[0020] Text density is defined as the ratio of the number of valid text characters in a web page, including Chinese characters, letters, and numbers, to the total number of characters on the web page; any web page with a text density lower than 75% or containing fewer than 100 valid characters will be ignored to ensure that the high quality standards of the dataset are maintained.

[0021] Furthermore, the data samples are deduplicated, including:

[0022] Step 11: Use a special word segmentation tool to segment the collected web page content. Considering that this dataset mainly deals with Chinese, the jieba word segmentation tool was selected. After word segmentation, in order to reduce the impact of stop words on text similarity evaluation, multiple stop word lists were combined to remove stop words from the word segmentation results.

[0023] Step 12, evaluate the importance of the word, including:

[0024] Apply the TF-IDF method to evaluate the importance of each word in the text;

[0025] First, calculate the term frequency TF, which is the number of times a word appears in a specific document divided by the total number of words in the document;

[0026] Then calculate the inverse document frequency IDF, which is the logarithm of the total number of documents containing the word;

[0027] Multiply a word's TF value by its IDF value. The higher the TF-IDF value, the more important the word is in the document.

[0028] Step 13, using the SimHash algorithm to evaluate and compare text similarity, including:

[0029] Perform MD5 hash operation on each word in the text to generate a 64-bit bit string;

[0030] Then, the hash value of each word is weighted according to its TF-IDF value, and the weighted hash values ​​of all words are accumulated;

[0031] On the accumulated result, each bit is judged. If the weighted sum is greater than zero, the corresponding bit is set to 1 in the SimHash value, otherwise it is set to 0. Each document is converted into a SimHash value of fixed length;

[0032] The Hamming distance is used to measure the difference between two SimHash values, that is, to compare the number of different characters in the same position of the two SimHash values. If the Hamming distance of the SimHash values ​​of two documents is less than or equal to 3, the two documents are highly similar in content and are marked as duplicate samples. Only the one with the longest number of characters among the similar samples is retained.

[0033] Step 14, using a heuristic regular expression method to detect and replace sensitive information, including: 1. The ID card number regular expression is: \b\d{15}|\d{17}[0-9Xx]\b; 2. The phone number starts with 1, the regular expression is: \b1[3-9]\d{9}\b; 3. The email regular expression is: \b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b; 4. The bank account information regular expression The formula is: \b\d{16}|\d{19}\b; 5. The regular expression for address information is: (?:province|city|district|county|road|street|number|building|unit|room)\b; Once personal privacy and sensitive information is identified, the information is replaced with special characters of the corresponding category, such as replacing the ID number with "[ID number]", the phone number with "[phone number]", the email with "[email]", the bank account information with "[bank account]", and the address information with "[address]";

[0034] Step 15, set up a self-duplicate content filter, paying special attention to the structural blocks of the web page including paragraphs ,title <h1>-< / h1> <h6>, and other containers ; Group web page content according to domain name, and group web page texts with the same domain name;

[0035] Subsequently, a frequency analysis is performed on each text block in each web page to count whether the text in each block appears in other pages. When a text block appears repeatedly in 70% of the same group of pages, the corresponding text block is identified as an advertisement or duplicate content; the text block will be marked and excluded from the final data set.

[0036] Further, step 21, text segmentation

[0037] Step 211, using a byte pair encoding word segmentation algorithm to construct a Chinese and English bilingual word segmenter and word list;

[0038] For all collected data, the text is decomposed into characters, and then the most frequently occurring character pairs are gradually merged; first, the vocabulary is initialized as the set of all characters in the text, and each character is regarded as a separate word or subword unit;

[0039] In a given text data, the frequency of occurrence of all adjacent character pairs or byte pairs is counted; the most frequently occurring character pairs are merged into a new unit; the new unit is then added to the vocabulary;

[0040] Repeat the counting and merging steps, merging the most frequent byte pairs in each iteration until the preset vocabulary size is reached; the vocabulary gradually expands from a single character to include more common byte pairs or character sequences;

[0041] Step 212, using the longest match algorithm to segment the data;

[0042] The word segmentation process starts from the beginning of the text and tries to match the longest string, which is a valid entry in the vocabulary;

[0043] Check the longest possible string starting from the current position and query whether the string exists in the vocabulary;

[0044] If a match is found, the current string is confirmed as a word unit, and then jump to the next character after the current string to continue repeating the process;

[0045] If at some point a longer match cannot be found, back off and try shorter strings until a valid vocabulary match is found; after tokenization, the data is converted into a sequence of tokens.

[0046] Step 213, using a chunking strategy to build input, concatenating all data from different samples together, and cutting them into a fixed maximum length as training input;

[0047] After word segmentation and chunking, the collected data is processed into training data with uniform word length for the next step of the training process.

[0048] Further, step 22, text encoding: facing the uniform length word sequence w=w1w2…w n , adopting the word-gram mapping method, converting the word-gram sequence into a dense vector format that can be effectively processed by the neural network at the input layer; including:

[0049] Each word w in the sequence i are converted into a vector representation e i ;

[0050] After receiving the input word vector sequence E = e1...e n After that, an RMS normalization step is used to normalize the input embeddings; the matrix is ​​described as follows:

[0051]

[0052] Among them, ⊙ represents element-by-element multiplication, Represents element-wise division.

[0053] Further, step 23, rotation position encoding: using the rotation position encoding at the attention layer at the input layer; comprising:

[0054] Rotational position encoding encodes position information by rotating each word embedding vector; it is expressed as follows:

[0055] RoPE(p,x)=Rotate(p)·x

[0056] Where p represents the position of the word unit, x is the word vector of the word unit, and Rotate(p) is a rotation matrix associated with the position p; the rotation matrix is ​​defined as follows:

[0057]

[0058] θ p is the rotation angle relative to position p, depending on the relative position of the word.

[0059] Further, step 24, setting the attention mechanism, residual network, normalization layer and fully connected network output layer and autoregressive training method; including:

[0060] Step 241, setting the residual network, the normalization layer and the fully connected network;

[0061] After each layer of attention mechanism, the model uses a residual network and layer normalization operation to normalize the output vector; described as a matrix operation:

[0062] H l =LayerNorm(H l-1 +Attn(O l-1 ))

[0063] Among them, H l is the hidden variable representation of the lth layer, O l-1 is the hidden variable output of the l-1th layer, Attn(·) is the attention network of the corresponding layer, and LayerNorm is the layer normalization operator. After the residual network, a fully connected network is used to model language knowledge, which can be described as the following matrix operation:

[0064] O l =LayerNorm(H l +GeGLU(WH l ))

[0065] Where W is the trainable weight parameter and GeGLU is the nonlinear activation function.

[0066] Step 242, setting the output layer and the autoregressive training method;

[0067] The output layer of the trained model is based on the representation O of the last layer. L , used to predict the conditional probability of the word at each position in the text sequence, expressed as:

[0068]

[0069] Among them, W e ∈R |V|×d Unsigned word vector matrix, |V| is the word table size, is the representation of the i-th word by the last encoding layer;

[0070] The trained model is a unidirectional language model that inputs a text sequence w in a natural reading order; the likelihood estimate of the text sequence is maximized; the model is trained to accurately predict the current word based on a given historical sequence; the process involves optimizing the following loss function:

[0071]

[0072] Where θ represents the model parameter and n represents the sequence length;

[0073] In order to optimize this loss function, the model training process adopts the gradient descent method, using the Adam optimizer for back propagation.

[0074] Further, step 25, selecting pre-training data, includes:

[0075] Calculate the overall data demand and predict the model capabilities; randomly select 4 billion data from the collected data as training data to train these models of different sizes, and record the changes in their training losses under different training resources to fit the relationship between training resources and model capabilities; according to the scaling rule:

[0076] L=a×logC+b

[0077] Where logC is the logarithmic form of the floating-point operation number of computing resources;

[0078] According to the available training resources, the training loss of the training target model is calculated to be about 2.7, which is marked with a red star. The relationship between the training resources C, the training model parameters N, and the amount of training data D is fitted as follows:

[0079] C≈6.03DN

[0080] The training resources and target model parameters are fixed. According to calculations, the available training data is about 500 billion words. In order to balance the capabilities brought by the two languages ​​and multiple types of data, the final training data is selected from the collected data according to the proportions shown in Table 2.

[0081] Among them, encyclopedia data in English data was reused 3 times, mathematics data was reused 2 times, conversation data was reused 2 times, science and technology data and book data were reused 1.5 times, half of the data was randomly sampled and reused, and the rest of the data were used once; in Chinese data, 25% of the web page data with a large number but poor overall quality was sampled, 50% of the book data was sampled, 70% of the conversation data was sampled, and all other data were used;

[0082] In order to improve the quality of sampling results in these types of Chinese, the data quality is scored, and the cross entropy loss function value of each data in the model is used as the score. The lower the score, the higher the quality.

[0083] Furthermore, step 3, instruction fine-tuning training, includes:

[0084] Step 31, data collection and processing for instruction fine-tuning;

[0085] Collect text data of directive language and interactive dialogue; the text data comes from online question-answering systems, chatbot logs, technical support dialogues, and other interactive platforms; the text data contains the user's specific instructions and the system's responses, showing clear instruction nature and interactive characteristics;

[0086] The collected directive language and interactive dialogues are organized into a multi-round dialogue format; the collected data has two formats, namely the command response format and the dialogue format; Table 6 shows how the three corresponding formats are converted into the multi-round dialogue format, and regular expressions are used to extract and format the information;

[0087] Table 6: Formatted dataset

[0088]

[0089] Use "You are a complete and secure conversational robot model. Complete the overall conversation content according to the user's needs, help the user complete the specified task, and pay attention to ensuring information security" as the [system instruction]; all data are converted into a unified form and then fine-tuned for instruction training;

[0090] Step 32, fine-tuning the instruction;

[0091] Instruction fine-tuning only trains the model response part and does not calculate other inputs; for a given sequence S = system instructions: s1s2...s n User: u1u2...u m Model: m1m2...m l , only the loss of the model output part is calculated, that is, the original loss function is modified to:

[0092]

[0093] The modified loss function ensures that the model only learns how to reply without modeling the user's input.

[0094] This application has the following effects:

[0095] 1. Resource efficiency and rapid deployment optimization

[0096] The lightweight bilingual large language model developed in the present invention significantly reduces the number of parameters of the model, thereby reducing the resource consumption required in the fine-tuning stage. Such a design not only makes the model run faster, but also greatly reduces the deployment and operation costs in cloud computing and edge computing environments. Therefore, the model is particularly suitable for occasions that require rapid deployment and maintenance, especially in resource-limited environments, such as mobile and embedded devices. In addition, due to its high efficiency and low resource requirements, this model shows higher practicality and feasibility in specific industry applications, such as finance, medical care, and law.

[0097] 2. Efficient bilingual processing function

[0098] The model trained by the method of the present invention significantly improves the bilingual processing ability of Chinese and English while maintaining a small scale. This means that the model can not only process mixed Chinese and English texts, but also effectively transfer and apply knowledge between different languages, improving its applicability in a multilingual environment. In addition, this method solves the data collection and quality issues of languages ​​with fewer resources such as Chinese, and improves the performance of the model in these languages, especially when processing professional terms and specific domain languages.

[0099] Improved flexibility and adaptability

[0100] The model trained by this method demonstrates extremely high flexibility and adaptability, and can be effectively adjusted according to the needs of different languages ​​and fields to achieve more accurate and efficient language processing in multilingual and multi-field environments. This flexibility enables the model to easily adapt to a variety of application scenarios, such as cross-cultural communication, international business, and multilingual information processing, greatly improving its breadth and effectiveness in practical applications. Overall, this patented technical solution not only solves the optimization problem of large-scale language models in specific fields, but also provides an innovative and practical solution for efficient language processing in multilingual environments. These technical effects have jointly promoted the development and widespread application of language model technology in practical industrial applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 A flowchart of the technical solution provided for the embodiment of the present application;

[0102] Figure 2 A data processing flow chart provided for an embodiment of the present application;

[0103] Figure 3 A model structure diagram provided for an embodiment of the present application;

[0104] Figure 4 A relationship diagram between training loss and training resources fitted according to the scaling rule provided in an embodiment of the present application;

[0105] Figure 5 The result diagram of the patented system of a municipal service center in a certain city provided in the embodiment of the present application. DETAILED DESCRIPTION

[0106] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0107] This application proposes an innovative solution to this challenge: a lightweight Chinese-English bilingual large language model training method. The aim is to significantly reduce the number of parameters and resource requirements during fine-tuning while maintaining efficient language processing capabilities. This lightweight model is not only easy to deploy and maintain quickly, but also shows better efficiency and practicality in specific field applications.

[0108] The following first introduces the embodiments of the present application in conjunction with the accompanying drawings.

[0109] Step 1: Training data collection and selection

[0110] Step 1.1 Data collection and preprocessing

[0111] In order to pre-train a large language model with basic language and knowledge capabilities, this method carefully collects a large amount of high-quality bilingual datasets in Chinese and English. For English data, this application uses the Pile dataset, an 800GB dataset containing 22 diverse sources such as web crawling, academic resources, and books. In addition, to improve the basic capabilities and factual reliability of the model, the weight of the corresponding content in the training process is increased by upsampling.

[0112] For Chinese pre-training data, data from the WuDao project and CBook, as well as open source materials from multiple Chinese websites, are integrated. The WuDao project covers 3 billion web pages with high text density, and these contents are deeply cleaned to improve data quality and filter out privacy-related information. Only 200GB of WuDao is available in an open version, and this part of the data covers multiple fields such as Wikipedia, news, and comments. Another key data source is CBook, a large-scale Chinese book corpus collected by the Natural Language Processing Laboratory of Fudan University. CBook contains the content of more than 100,000 books, and the texts of these books are extracted from multiple formats (such as PDF, MOBI, and EPUB). Since the above data sets total about 300GB, they are not enough to train large-scale language models. Due to the rapid development of the Internet environment in recent years, web page texts usually provide sufficient data. The present invention also collects a large amount of Chinese text from various websites. To ensure the consistency, diversity, and quality of these self-collected data. This patent carries out a strict preprocessing process, including special tag removal, sensitive data cleaning, data deduplication, and comprehensive screening to remove various low-quality content, such as advertisements and inappropriate website materials.

[0113] Use the web page parser Beautiful Soup to accurately extract and clean the web page text content, eliminate useless HTML, CSS and JS identifiers, emoticons and other abnormal symbols in the web page, and ensure the fluency and usability of the text content. Considering that the purpose is to train a single-modal plain text language model, we focus on extracting plain text data from the web page, traversing and analyzing all text-related nodes of the web page, including the title ( <h1> to< / h1> <h6>),paragraph( ),span( ),piece( ) and lists ( ), using the powerful parsing ability of Beautiful Soup to extract the text content. In order to ensure the purity of the extraction results, images are ignored during the extraction process ( ) and Video( <video>) and other non-text elements.

[0114] Since web page content may come from a variety of different websites, the character encoding of each site may be different, so character encoding detection and processing are performed; the chardet library is used to identify the encoding of the extracted text in order to accurately understand and process the data. The specific encoding will be identified, and in order to facilitate subsequent data processing and analysis, the original data will be converted to UTF-8 encoding; during the transcoding process, if a byte sequence that cannot be recognized or cannot be correctly converted is encountered, including special or rare character encodings, mixed encodings of text, or non-text content, the unconverted content will be discarded to maintain the high quality and consistency of the data set.

[0115] In order to unify the format of the text crawled from the web page, formatting is performed. Delete the extra spaces, tabs, paragraph breaks and other invisible symbols before and after each paragraph in the text content. Special space characters, including Unicode U+00A0 and U+200B, are replaced with standard spaces. In view of the fact that the line terminators between different operating systems may be inconsistent, '\n' is selected as a unified line terminator to replace all the original line terminators in the text. For other invisible characters, such as carriage return (U+000D), soft line break (U+00AD) and other control characters, they are also replaced uniformly. In order to avoid the replacement operation causing multiple consecutive spaces in the text, the consecutive spaces are merged into a single space through a regular expression. Since these web page data are only for Chinese web pages, and spaces between characters should not usually appear in Chinese text, regular expressions are used to delete unnecessary spaces between Chinese characters. It should be noted that this application retains traditional Chinese characters, so that the model can obtain learning materials from classical literature or poetry.

[0116] The initial quality assessment of web page content is carried out by calculating the text density. The text density is defined as the ratio of the number of valid text characters in the web page, including Chinese characters, letters, and numbers, to the total number of characters in the web page (including HTML tags, JavaScript code, CSS styles, etc.). In order to accurately determine which text density threshold can effectively screen out high-quality data, an assessment is carried out. Specifically, 1000 web page samples are randomly selected from the collected data, and the samples are evaluated to determine whether each web page contains sufficient information and mark the quality. Calculate the text density of the samples, and traverse different thresholds from 10% to 90%, and calculate the F1 score for classifying the web pages at each threshold to evaluate the accuracy of the classification effect. Through this process, a text density threshold of 75% can most effectively judge the data quality of the web page, and 75% is used as the text density threshold. During the evaluation and threshold search process, special types of web pages, such as comic web pages containing pictures and their short descriptions, or minimalist web pages containing only titles, although these web pages have short content but may still be rich in information. Therefore, using only text density as a filtering criterion is not always effective in these cases. Considering these factors, a method combining text density and the number of valid characters is used for preliminary quality filtering. Finally, it is determined that any web page with a text density lower than 75% or containing less than 100 valid characters will be ignored to ensure the maintenance of high-quality standards for the dataset.

[0117] Since the WuDao project also extracts content from web pages, and some websites will repost the same information, in order to reduce data duplication and improve data quality, duplicate removal is performed on the data samples. In the data preprocessing stage, first use a dedicated word segmentation tool to segment the collected web page content. Considering that this dataset mainly processes Chinese, the jieba word segmentation tool is selected, which can effectively identify the word boundaries in Chinese. After word segmentation, in order to reduce the impact of stop words (such as frequently appearing words like "de", "he", etc. that contribute little to understanding the text theme) on the text similarity evaluation, combine multiple stop word lists, such as the Harbin Institute of Technology stop word list and the Baidu stop word list, and remove the stop words from the word segmentation results.

[0118] The TF-IDF method is applied to evaluate the importance of each word in the text. First, calculate the term frequency (TF), which is the number of times a word appears in a specific document divided by the total number of words in the document; then calculate the inverse document frequency (IDF), which is the logarithm reciprocal of the total number of documents based on the number of documents containing the word. Multiply the TF value of a word by its IDF value, and the higher the resulting TF-IDF value, the more important the current word is in the document. The SimHash algorithm is used to evaluate and compare text similarity, and the steps include:

[0119] Perform MD5 hashing on each word in the text to generate a 64-bit bit string;

[0120] Then, the hash value of each word is weighted according to its TF-IDF value, and the weighted hash values ​​of all words are accumulated.

[0121] On the accumulated result, each bit is judged. If the weighted sum is greater than zero, the corresponding bit is set to 1 in the SimHash value, otherwise it is set to 0. Each document is converted into a SimHash value of fixed length.

[0122] The Hamming distance is used to measure the difference between two SimHash values, that is, to compare the number of different characters in the same position of two SimHash values; if the Hamming distance of the SimHash values ​​of two documents is less than or equal to 3, the two documents are highly similar in content and are marked as duplicate samples. Only the one with the longest number of characters among the similar samples is retained to optimize the content quality and uniqueness of the data set. This series of sophisticated processing steps ensures the accuracy and availability of the data, providing a high-quality foundation for subsequent data analysis and model training.

[0123] Web pages often contain personal privacy and sensitive information. Sensitive vocabulary and heuristic methods are used to detect and filter information. First, this patent collects filtering vocabulary from different sources, which are maintained by government agencies, social organizations or online communities. Then, consult relevant laws and regulations to extract words that are explicitly prohibited, including personal privacy and sensitive information. In order to identify and process this information, a heuristic regular expression method is used for detection, including: 1. ID card number: ID card number usually has a specific format, usually 18 digits (or 15 digits in the old version), of which the first 17 digits are numbers, and the last digit may be a number or the letter 'X'. The regular expression is: \b\d{15}|\d{17}[0-9Xx]\b; 2. Phone number: The format of the phone number may vary by country or region. Taking Chinese mobile phone numbers as an example, they are usually 11 digits long and start with 1. The regular expression is: \b1[3-9]\d{9}\b; 3. Email: The format of the email address is usually username@domain name. The regular expression is: \b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b; 4. Bank account information: Bank account information has various formats and generally contains a certain length of digital sequence. Chinese bank accounts are generally 16 or 19 digits long. The regular expression is: \b\d{16}|\d{19}\b; 5. Address information: Address information is relatively complex and involves multiple formats, including specific address information such as province, city, district, and street. Regular expressions are used to identify some common keywords, such as province, city, district, street, etc., and format matching text fragments containing specific address keywords. The regular expression is as follows: (?:province|city|district|county|road|street|number|building|unit|room)\b; once personal privacy and sensitive information is identified, the information is replaced with special characters of the corresponding category, such as replacing the ID number with "[ID number]", the phone number with "[phone number]", the email with "[email]", the bank account information with "[bank account]", and the address information with "[address]" to protect user privacy. This process ensures that all identified sensitive information will not be leaked while retaining the structure and readability of the text.

[0124] Web pages often contain duplicate advertising content, especially on multiple pages under the same domain. In order to effectively filter out duplicate advertising content under the same domain, set up a self-duplicate content filter, paying special attention to the structural blocks of the web page, including paragraphs. ,title <h1>-< / h1> <h6>, and other containers ; Web page blocks often contain repeated advertising statements, which affects data quality. Group web page content according to domain names, and group web page texts with the same domain name;

[0125] Subsequently, the frequency of occurrence of each text block in each web page is analyzed, and statistics are given to see whether the text in each block appears in other pages. When a text block appears repeatedly in 70% of the same set of pages, the corresponding text block is identified as an advertisement or duplicate content; the text block will be marked and excluded from the final data set. In this way, the patent can accurately filter out repeated advertising content. In addition, repeated advertising text blocks detected under different domain names are used as training data, and the Latent Dirichlet Allocation (LDA) algorithm is used to construct an advertising phrase vocabulary. When training the LDA model, the entire process involves initializing the model parameters and gradually refining these parameters through multiple iterations to achieve convergence.

[0126] First, the model parameters, including the topic distribution for each document and the word distribution for each topic, are randomly initialized. For each document in the model, each word is initially randomly assigned to a topic. During the iteration process, the initial assignment is continuously updated based on the probabilities calculated by the model to reflect more accurate topic attribution. The model calculates and reallocates each word to the topic that maximizes its probability of appearing in the current document, which involves combining the probability of the current word under the current topic and the proportion of the current topic in the entire document. Subsequently, based on the new assignment, the topic distribution of the document and the word distribution of the topic are updated, and this update reflects the change in the corresponding topics of each word in all documents. This process is repeated, and the topic assignment and update parameters are adjusted based on the previous results each time until the topic assignment is stable or the preset number of iterations is reached, so as to ensure that the topic distribution of each document and the vocabulary composition of each topic have reached the data-driven optimal state. After training the LDA model, the model outputs the word distribution of each topic, which reflects the most relevant words for each topic. Look at the topics that are directly related to the ad content and select the words with high weights; these high-weight words are usually keywords that appear frequently in ads, such as "discount", "free", "limited time", etc. Finally, the high-weight words are organized into a list as a key advertising word list; the key advertising word list will be used to identify advertisements in text content. When the appearance rate of advertising phrases in the text content exceeds 50%, the content will be re-filtered.

[0127] Finally, the data quality needs to be evaluated. This patent adopts a manual sampling evaluation method to evaluate text quality. The quality of the overall text data is evaluated by extracting some text samples and manually reviewing them. First, this patent adopts random sampling combined with stratified sampling to ensure the accuracy and reliability of the evaluation results. Afterwards, for the sampled text samples, clear evaluation standards and rules are formulated, such as clarity, logic, coherence, etc., and provided to the evaluators for understanding and execution. Finally, based on the results of manual sampling evaluation, problems and deficiencies in the data set can be identified. According to the evaluation results, take corresponding quality improvement measures.

[0128] Through this series of precise data processing, this patent ultimately built a huge Chinese text corpus of approximately 1.8TB, covering many different fields from novels to law. Figure 2 The entire data processing process is presented in detail. The richness and diversity of these data provide a comprehensive and balanced foundation for the training of this patented model to support the efficient learning and application of the model in a multilingual environment.

[0129] The final data results obtained are shown in Table 1.

[0130] Table 1 Data Overview

[0131]

[0132] Step 2: Pre-training

[0133] With the rapid spread of the global Internet, a huge amount of text information has accumulated on the Internet, covering various forms from books in public libraries to posts on social media platforms. In these data, various languages ​​show different statistical characteristics, especially in terms of data quantity and quality. In such an environment, Chinese and English, as one of the most widely used languages ​​in the world, have their relevant high-quality data widely collected and made public. Therefore, the present invention specially selects high-quality Chinese and English text data for model training in the initial stage, so as to lay a solid foundation for building a language model and improve the ability to process language.

[0134] Step 2.1 Text segmentation

[0135] In the field of natural language processing, text segmentation plays a vital role, especially in the training of large language models. The core goal of word segmentation is to convert complex text information into smaller processing units, namely tokens.

[0136] The byte pair encoding word segmentation algorithm is used to build bilingual word segmenters and word lists for Chinese and English. The core of the byte pair encoding word segmentation algorithm is an iterative character merging mechanism. For all collected data, the text is first decomposed into characters, and then the most frequently occurring character pairs are gradually merged; specifically, the vocabulary is first initialized as the set of all characters in the text, and each character is regarded as a separate word or subword unit; in the given text data, the frequency of occurrence of all adjacent character pairs or byte pairs is counted; the most frequently occurring character pairs are merged into a new unit; the new unit is then added to the vocabulary. For example, if "es" is the most common character pair, then "es" will be merged into a new unit "es". Repeat the counting and merging steps, merging the most frequent byte pairs each iteration until the preset vocabulary size is reached; the vocabulary gradually expands from a single character to include more common byte pairs or character sequences; the key to this method is that it does not rely on a predefined vocabulary, but dynamically generates a vocabulary based on actual data, and is a language-independent data-driven word segmentation algorithm. After the vocabulary is built, the data is tokenized; the tokenization process uses the longest match algorithm; starting from the beginning of the text, an attempt is made to match the longest string, which is a valid entry in the vocabulary; the longest possible string starting from the current position is checked and whether this string exists in the vocabulary; if a match is found, the current string is confirmed as a word unit, and then the algorithm jumps to the next character after the current string and continues to repeat the process; if a longer match cannot be found at some point, the algorithm backs up and tries a shorter string until a valid vocabulary match is found; after tokenization, the data is converted into a sequence of word units.

[0137] In order to be able to perform batch processing when training the model, it is necessary to form input data of the same length. However, due to the different lengths of the original input data, the previous method requires padding or truncation of each data. In order to improve the input efficiency of the model and reduce the data waste caused by excessive padding, the present invention adopts a block strategy to form the input; this strategy is to splice all the data from different samples together and cut them into a fixed maximum length (for example, the present invention uses 4096 words) as training input. This cutting method can effectively utilize the input capacity of the model. Secondly, it enables the model to process longer text sequences, thereby better understanding the long-distance dependencies in the text and reducing the degradation of model performance that may be caused by filling a large amount of irrelevant data.

[0138] After word segmentation and chunking, the collected data is processed into training data with uniform word length for the next step of the training process.

[0139] Step 2.2 Text encoding

[0140] Faced with a uniform length word sequence w=w1w2…w input to the model n , the present invention adopts the word-unit mapping method to convert the word-unit sequence into a dense vector format that can be effectively processed by the neural network at the input layer. Specifically, each word w in the sequence i are converted into a vector representation e i ;

[0141] After receiving the input word vector sequence E = e1...e n After that, an RMS normalization step is used to normalize the input embeddings; the matrix is ​​described as follows:

[0142]

[0143] Among them, ⊙ represents element-by-element multiplication, Represents element-wise division.

[0144] Step 2.3 Rotational Position Encoding

[0145] Unlike the large language model proposed by previous studies, the present invention does not use learned position encoding in the input layer, but uses rotational position encoding in the attention layer, which is a key adjustment based on the characteristics of the model trained by the present invention. Since the Transformer framework itself cannot naturally perceive position offsets, and the learned position encoding used by predecessors can only process fixed-length inputs and is difficult to extrapolate, the present invention compensates for this by introducing rotational position encoding with absolute relative position information.

[0146] Rotational position encoding refers to incorporating the position information of a word into a vector by means of a rotational transformation. This method is different from the traditional method of adding a position vector. Rotational position encoding encodes the position information by rotating each word embedding vector; it is expressed as follows:

[0147] RoPE(p,x)=Rotate(p)·x

[0148] Where p represents the position of the word, x is the word vector of the word, and Rotate(p) is a rotation matrix associated with the position p; the rotation matrix is ​​defined as follows:

[0149]

[0150] θ p is the rotation angle associated with position p, which depends on the relative position of the word. In this way, the rotational position encoding allows the model to capture the relative position information in the sequence, allowing the model to more accurately understand the context and semantic relationships in the text.

[0151] The advantage of rotational position encoding is that it can provide a more natural and dynamic way to handle position information, especially when processing long sequences. This method not only improves the model's ability to understand long-distance dependencies, but also enables the model to more flexibly handle text sequences of different lengths due to its relative position encoding method.

[0152] Step 2.3 Attention Mechanism

[0153] FlashAttention2 is used as the attention mechanism to accelerate and optimize the training process of large language models. The core goal of FlashAttention is to solve the computational efficiency and resource consumption problems faced by traditional attention mechanisms when processing large-scale data.

[0154] FlashAttention optimizes attention calculations at the hardware level. Through algorithmic improvements, it significantly reduces non-matrix multiplication (non-matmul) floating-point operations (FLOPs), thereby improving overall computational efficiency. Specifically, FlashAttention implements the following optimization measures:

[0155] 1. Work division optimization: FlashAttention performs optimal work division for attention calculations to reduce memory usage and reduce computing requirements. This optimization allows the model to maintain high performance while reducing dependence on hardware resources.

[0156] 2. Parallel processing: By parallelizing attention calculations across different thread blocks, FlashAttention can perform operations on multiple processing units simultaneously, further improving processing speed.

[0157] 3. Appropriate work distribution: In a multi-core processing environment, FlashAttention effectively distributes computing tasks to different processing cores (such as GPU warps), ensuring that each core is loaded evenly and avoiding single-point computing bottlenecks.

[0158] Use the FlashAttention implementation from the flash-attn package.

[0159] Step 2.4 Residual network, normalization layer and fully connected network

[0160] After each layer of attention mechanism, the model uses a residual network and layer normalization operation to normalize the output vector to improve the stability of the model; described as a matrix operation:

[0161] H l =LayerNorm(H l-1 +Attn(O l-1 ))

[0162] Among them, H l is the hidden variable representation of the lth layer, O l-1 is the hidden variable output of the l-1th layer, Attn(·) is the attention network of the corresponding layer, and LayerNorm is the layer normalization operator. After the residual network, a fully connected network is used to model language knowledge, which can be described as the following matrix operation:

[0163] O l =LayerNorm(H l +GeGLU(WH l ))

[0164] Where W is a trainable weight parameter, and GeGLU is a nonlinear activation function. The present invention uses a total of 32 layers of neural networks. Figure 3 shown.

[0165] Step 24, output layer and autoregressive training method;

[0166] The output layer of the trained model is based on the representation O of the last layer. L , used to predict the conditional probability of the word at each position in the text sequence, expressed as:

[0167]

[0168] Among them, W e ∈R |V|×d Unsigned word vector matrix, |V| is the word table size, is the representation of the i-th word by the last encoding layer;

[0169] The model trained by the present invention is a unidirectional language model, which inputs a text sequence w in a natural reading order; adopts the traditional language model objective, that is, maximizing the likelihood estimate of the text sequence; the model is trained to accurately predict the current word based on a given historical sequence; the process involves optimizing the following loss function:

[0170]

[0171] Where θ represents the model parameter and n represents the sequence length;

[0172] In order to optimize this loss function, the model training process uses the gradient descent method, using the Adam optimizer for back propagation. The Adam optimizer is an efficient gradient descent variant that combines the characteristics of momentum and adaptive learning rate to achieve faster convergence and higher training stability. This optimization strategy enables the model to adjust parameters more accurately during the learning process to adapt to complex and changing language patterns, ultimately achieving the ability to deeply understand and generate language.

[0173] Step 25: Pre-training data selection

[0174] Due to limited computing resources and the large size of the target model, in order to ensure the stability of the overall training process, the overall data requirements are calculated and the model capabilities are predicted. For the same model architecture, different hyperparameters such as the number of neural network layers and model dimension sizes are set to build models with different sizes of 10 million parameters, 50 million parameters, 150 million parameters, 350 million parameters and 500 million parameters. 4 billion (about 10 billion words) of data are randomly selected from the collected data as training data to train these models of different sizes, and their changes in training losses under different training resources are recorded to fit the relationship between training resources and model capabilities; according to the scaling law:

[0175] L=a×logC+b

[0176] Where logC is the logarithmic form of the floating-point operations of computing resources (Log FLOPs). The present invention fits the relationship between training loss and training resources when using model training, such as Figure 4 shown.

[0177] According to the available training resources, the training loss of the training target model is calculated to be about 2.7, which is marked with a red star in the figure. In addition, through the above experiments, the relationship between the training resources C, the training model parameters N, and the amount of training data D is also fitted:

[0178] C≈6.03DN

[0179] The training resources and target model parameters used in the present invention are fixed, and according to the formula, it can be calculated that the available training data is about 500 billion words. In order to balance the capabilities brought by two languages ​​and multiple types of data, the final training data is screened out from the proportions in Table 2 of the collected data. Among them, the encyclopedia data in the English data is reused 3 times, the mathematics data is reused 2 times, the dialogue data is reused 2 times, the science and technology data and the book data are reused 1.5 times, half of the data is randomly sampled and reused, and the remaining types of data are used once. In the Chinese data, 25% of the web page data with a large number but poor overall quality is sampled, 50% of the book data is sampled, and 70% of the dialogue data is sampled, and all other data are used. In order to improve the quality of the sampling results under these types of Chinese, the 10 million-level model trained in this step is used to score the data quality. Specifically, the cross entropy loss function value of each data on the model is used as the score, and the lower the score, the higher the quality.

[0180] Table 2. Ratio and quantity of data trained for pre-training

[0181]

[0182] Step 2.6 Pre-training

[0183] As shown in Table 2, the number of data samples in Chinese and English is the same. In order to ensure the balance of bilingual capabilities during training, each batch is input using a crossover of Chinese and English data, that is, one sample is taken from both the English data and the Chinese data until the training batch size is reached and then input into the model for training.

[0184] In terms of basic English ability, the model pre-trained in this patent performs well in two English large-scale language model benchmark tests MMLU and AGIEval, surpassing other larger-scale models. The test results are shown in Table 3.

[0185] Table 3 Evaluation results of the model trained by this patent method on the English benchmark.

[0186]

[0187] In terms of Chinese language processing capabilities, the model pre-trained by this method achieved outstanding results on two major benchmark tests of Chinese language models, CMMLU and C-Eval, and even outperformed other larger models. This fully demonstrates the efficiency and accuracy of this patented method in processing Chinese tasks. Table 4 lists the results of this patented model in the Chinese benchmark evaluation.

[0188] Table 4 Evaluation results of the model trained by this patent method on Chinese benchmarks

[0189]

[0190] Step 3: Instruction fine-tuning training

[0191] After pre-training, instruction fine-tuning training is used to optimize the performance of the model in specific tasks and scenarios to ensure that it can adapt to complex practical application needs. The entire process of instruction fine-tuning training covers multiple steps such as data collection, data formatting, and instruction fine-tuning training.

[0192] Step 31: Data collection and processing for instruction fine-tuning

[0193] Unlike the data collection work in pre-training data, which focuses on obtaining a wide range of high-quality text datasets, in the instruction fine-tuning stage, text data of directive language and interactive dialogues are collected; these data are crucial to improving the model's ability to understand and execute instructions. The text data comes from online question-and-answer systems, chatbot logs, technical support conversations, and other interactive platforms. The text data contains the user's specific instructions and the system's responses, showing clear instruction nature and interactive features. For example, a user may ask "How do I reset my password?" and then the system provides detailed steps and guidance. Such conversations not only provide a rich interactive context, but also cover specific instructions in a variety of scenarios, thereby providing the model with an opportunity to learn how to understand and respond to these instructions. As shown in Table 5, this patent collects a variety of types of instruction fine-tuning datasets in both Chinese and English.

[0194] Table 5 Instruction fine-tuning dataset collected by this patent method

[0195]

[0196] Due to different data sources, the collected data usually has different formats. In order to ensure the uniformity of the data format, the collected directive language and interactive dialogues are organized into a multi-round dialogue format. The purpose of this step is to enable the model to learn a specific dialogue structure and understand the instruction logic. In this way, the model can learn how to extract key information from the context and respond accordingly to the instructions. This formatted data not only helps the model understand specific instructions, but also trains the model's ability to recognize and respond to similar instructions, thereby achieving efficient instruction execution in different application scenarios. Specifically, the collected data usually has two formats, namely the instruction response format and the dialogue format; Table 6 shows how the three corresponding formats are converted into a multi-round dialogue format. In the implementation, regular expressions are used to extract and format information.

[0197] Table 6: Formatted dataset

[0198]

[0199] Use "You are a complete and secure conversational robot model. Complete the overall conversation content according to the user's needs, help the user complete the specified task, and pay attention to ensuring information security" as the [system instruction]. All data is converted into a unified form and then fine-tuned for instruction training.

[0200] Step 32 Instruction fine-tuning

[0201] Similar to the pre-training in step 2, instruction fine-tuning also uses the Adam optimizer to train the autoregressive loss on the model. The difference is that the autoregressive loss of pre-training is calculated for all input words, while instruction fine-tuning only trains the model response part and does not calculate other inputs; for a given sequence S = system instructions: s1s2...s n User: u1u2...u m Model: m1m2...m l , only the loss of the model output part is calculated, that is, the original loss function is modified to:

[0202]

[0203] The modified loss function ensures that the model only learns how to reply without modeling the user's input. After the training is completed, the performance of the model is comprehensively evaluated and analyzed. This includes evaluating the performance of the model on different types of texts and tasks, as well as ensuring its stability and reliability through a series of tests and verifications. The training results show that the model of this patent, which has been fine-tuned with instructions, performs well in handling various natural language processing tasks, especially in understanding complex instructions and conducting efficient interactions. Through this series of training and evaluation, this patent has successfully improved the performance and applicability of the model in practical applications. Specifically, this patent was evaluated on Chinese C-Eval, and the results are shown in Table 4. After fine-tuning the instructions, the model of this patent has greatly improved its capabilities and surpassed various ultra-large-scale language models.

[0204] Table 4 Benchmark evaluation results of the model after fine-tuning the instructions of the patent method

[0205]

[0206] In summary, the instruction fine-tuning training process is an important part of this patent. Through this process, this patent not only enhances the model's ability to understand various language structures, but also improves its adaptability and reliability in practical applications.

[0207] Adaptation case of a city’s government services: The model trained by this method is adaptively trained for a city’s government services, providing citizens with government services such as policy interpretation in the form of dialogue, reducing municipal service expenses and improving municipal service efficiency.

[0208] Figure 5 It displays relevant questions and answers about a city and successfully provides the source of reference documents, thus realizing a credible and reliable government affairs question and answer system.

[0209] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application. < / h6> < / video> < / h6> < / h6> < / video> < / h6>

Claims

1. A lightweight Chinese-English bilingual large language model training method, characterized in that: The method comprises: Step 1: Collect and preprocess Chinese and English training data; preprocessing includes character encoding detection and processing, preliminary quality assessment of web page content, and deduplication of data samples; Step 2: Pre-training; including: Step 21, perform text segmentation; Step 22, text encoding: Faced with the uniform length word sequence w=w1w2…w input to the model n , adopting the word-gram mapping method, the word-gram sequence is converted into a dense vector format that can be effectively processed by the neural network at the input layer; Step 23, rotational position encoding: the rotational position encoding in the attention layer is used in the input layer; rotational position encoding refers to integrating the position information of the word unit into the vector in a rotational transformation manner; Step 24, setting the attention mechanism, residual network, normalization layer, fully connected network output layer and autoregressive training method; wherein FlashAttention2 is used as the attention mechanism; Step 25, select pre-training data; Step 26, pre-training; each batch is input using a cross-pattern of Chinese and English data, that is, one sample is taken from the English data and one from the Chinese data, until the training batch size is reached and then input into the model for training. Step 3: Instruct fine-tuning training.

2. The method according to claim 1, characterized in that Step 1, character encoding detection and processing, including: Use the web page parser Beautiful Soup to accurately extract and clean the text content of web pages, eliminate useless HTML, CSS and JS identifiers, emoticons and other abnormal symbols in the web pages, extract plain text data from the web pages, traverse and analyze all text-related nodes of the web pages, including titles <h1> to< / h1> <h6>,paragraph ,span ,piece and list , using Beautiful Soup to extract the text content. Images are ignored during the extraction process and Video <video> Non-text elements;< / video> Detect and process character encodings; use the chardet library to identify the encoding of the extracted text; identify the specific encoding and convert the original data into UTF-8 encoding; during the transcoding process, if you encounter a byte sequence that cannot be recognized or cannot be correctly converted, including special or rare character encodings, mixed encodings of text, or non-text content, then discard the content that cannot be converted.

3. The method according to claim 1, characterized in that Initial quality assessment of web page content, including: Perform a preliminary quality assessment of web page content by calculating text density; Text density is defined as the ratio of the number of valid text characters in a web page, including Chinese characters, letters, and numbers, to the total number of characters on the web page; any web page with a text density lower than 75% or containing fewer than 100 valid characters will be ignored to ensure that the high quality standards of the dataset are maintained.

4. The method according to claim 1, characterized in that: De-duplicate the data samples, including: Step 11: Use a special word segmentation tool to segment the collected web page content. Considering that this dataset mainly deals with Chinese, the jieba word segmentation tool was selected. After word segmentation, in order to reduce the impact of stop words on text similarity evaluation, multiple stop word lists were combined to remove stop words from the word segmentation results. Step 12, evaluate the importance of the word, including: Apply the TF-IDF method to evaluate the importance of each word in the text; First, calculate the term frequency TF, which is the number of times a word appears in a specific document divided by the total number of words in the document; Then calculate the inverse document frequency IDF, which is the logarithm of the total number of documents containing the word; Multiply a word's TF value by its IDF value. The higher the TF-IDF value, the more important the word is in the document. Step 13, using the SimHash algorithm to evaluate and compare text similarity, including: Perform MD5 hash operation on each word in the text to generate a 64-bit bit string; Then, the hash value of each word is weighted according to its TF-IDF value, and the weighted hash values ​​of all words are accumulated; On the accumulated result, each bit is judged. If the weighted sum is greater than zero, the corresponding bit is set to 1 in the SimHash value, otherwise it is set to 0. Each document is converted into a SimHash value of fixed length; The Hamming distance is used to measure the difference between two SimHash values, that is, to compare the number of different characters in the same position of the two SimHash values. If the Hamming distance of the SimHash values ​​of two documents is less than or equal to 3, the two documents are highly similar in content and are marked as duplicate samples. Only the one with the longest number of characters among the similar samples is retained. Step 14, using a heuristic regular expression method to detect and replace sensitive information, including:

1. The ID card number regular expression is: \b\d{15}|\d{17}[0-9Xx]\b; 2. The phone number starts with 1, the regular expression is: \b1[3-9]\d{9}\b; 3. The email regular expression is: \b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b; 4. The bank account information regular expression The formula is: \b\d{16}|\d{19}\b; 5. The regular expression for address information is: (?:province|city|district|county|road|street|number|building|unit|room)\b; Once personal privacy and sensitive information is identified, the information is replaced with special characters of the corresponding category, such as replacing the ID number with "[ID number]", the phone number with "[phone number]", the email with "[email]", the bank account information with "[bank account]", and the address information with "[address]"; Step 15, set up a self-duplicate content filter, paying special attention to the structural blocks of the web page including paragraphs ,title <h1>-< / h1> <h6>, and other containers ; Group web page content according to domain name, and group web page texts with the same domain name; Subsequently, a frequency analysis is performed on each text block in each web page to count whether the text in each block appears in other pages. When a text block appears repeatedly in 70% of the same group of pages, the corresponding text block is identified as an advertisement or duplicate content; the text block will be marked and excluded from the final data set.

5. The method according to claim 1, characterized in that Step 21: Text segmentation Step 211, using a byte pair encoding word segmentation algorithm to construct a Chinese and English bilingual word segmenter and word list; For all collected data, the text is decomposed into characters, and then the most frequently occurring character pairs are gradually merged; first, the vocabulary is initialized as the set of all characters in the text, and each character is regarded as a separate word or subword unit; In a given text data, the frequency of occurrence of all adjacent character pairs or byte pairs is counted; the most frequently occurring character pairs are merged into a new unit; the new unit is then added to the vocabulary; Repeat the counting and merging steps, merging the most frequent byte pairs in each iteration until the preset vocabulary size is reached; the vocabulary gradually expands from a single character to include more common byte pairs or character sequences; Step 212, using the longest match algorithm to segment the data; The word segmentation process starts from the beginning of the text and tries to match the longest string, which is a valid entry in the vocabulary; Check the longest possible string starting from the current position and query whether the string exists in the vocabulary; If a match is found, the current string is confirmed as a word unit, and then jump to the next character after the current string to continue repeating the process; If at some point a longer match cannot be found, back off and try shorter strings until a valid vocabulary match is found; after tokenization, the data is converted into a sequence of tokens. Step 213, using a chunking strategy to build input, concatenating all data from different samples together, and cutting them into a fixed maximum length as training input; After word segmentation and chunking, the collected data is processed into training data with uniform word length for the next step of the training process.

6. The method according to claim 1, characterized in that Step 22, text encoding: Faced with the uniform length word sequence w=w1w2…w input to the model n , adopting the word-gram mapping method, converting the word-gram sequence into a dense vector format that can be effectively processed by the neural network at the input layer; including: Each word w in the sequence i are converted into a vector representation e i ; After receiving the input word vector sequence E = e1...e n After that, an RMS normalization step is used to normalize the input embeddings; the matrix is ​​described as follows: Among them, ⊙ represents element-by-element multiplication, Represents element-wise division.

7. The method according to claim 1, characterized in that Step 23, rotation position encoding: using the rotation position encoding at the attention layer at the input layer; including: Rotational position encoding encodes position information by rotating each word embedding vector; it is expressed as follows: RoPE(p,x)=Rotate(p)·x Where p represents the position of the word unit, x is the word vector of the word unit, and Rotate(p) is a rotation matrix associated with the position p; the rotation matrix is ​​defined as follows: θ p is the rotation angle relative to position p, depending on the relative position of the word.

8. The method according to claim 1, characterized in that Step 24, setting the attention mechanism, residual network, normalization layer, fully connected network output layer and autoregressive training method; including: Step 241, setting the residual network, the normalization layer and the fully connected network; After each layer of attention mechanism, the model uses a residual network and layer normalization operation to normalize the output vector; described as a matrix operation: H l =LayerNorm(H l-1 +Attn(O l-1 )) Among them, H l is the hidden variable representation of the lth layer, O l-1 is the hidden variable output of the l-1th layer, Attn(·) is the attention network of the corresponding layer, and LayerNorm is the layer normalization operator. After the residual network, a fully connected network is used to model language knowledge, which can be described as the following matrix operation: O l =LayerNorm(H l +GeGLU(WH l )) Where W is the trainable weight parameter and GeGLU is the nonlinear activation function. Step 242, setting the output layer and the autoregressive training method; The output layer of the trained model is based on the representation O of the last layer. L , used to predict the conditional probability of the word at each position in the text sequence, expressed as: Among them, W e ∈R |V|×d Unsigned word vector matrix, |V| is the word table size, is the representation of the i-th word by the last encoding layer; The trained model is a unidirectional language model that inputs a text sequence w in a natural reading order; the likelihood estimate of the text sequence is maximized; the model is trained to accurately predict the current word based on a given historical sequence; the process involves optimizing the following loss function: Where θ represents the model parameter and n represents the sequence length; In order to optimize this loss function, the model training process adopts the gradient descent method, using the Adam optimizer for back propagation.

9. The method according to claim 1, characterized in that: Step 25, selecting pre-training data; including: Calculate the overall data demand and predict the model capabilities; randomly select 4 billion data from the collected data as training data to train these models of different sizes, and record the changes in their training losses under different training resources to fit the relationship between training resources and model capabilities; according to the scaling rule: L=a×logC+b Where logC is the logarithmic form of the floating-point operation number of computing resources; According to the available training resources, the training loss of the training target model is calculated to be about 2.7, which is marked with a red star. The relationship between the training resources C, the training model parameters N, and the amount of training data D is fitted as follows: C≈6.03DN The training resources and target model parameters are fixed. According to calculations, the available training data is about 500 billion words. In order to balance the capabilities brought by the two languages ​​and multiple types of data, the final training data is selected from the collected data in the proportions shown in Table 2. Among them, encyclopedia data in English data was reused 3 times, mathematics data was reused 2 times, conversation data was reused 2 times, science and technology data and book data were reused 1.5 times, half of the data was randomly sampled and reused, and the rest of the data were used once; in Chinese data, 25% of the web page data with a large number but poor overall quality was sampled, 50% of the book data was sampled, 70% of the conversation data was sampled, and all other data were used; In order to improve the quality of sampling results in these types of Chinese, the data quality is scored, and the cross entropy loss function value of each data in the model is used as the score. The lower the score, the higher the quality.

10. The method according to claim 1, characterized in that Step 3: instruction fine-tuning training; including: Step 31, data collection and processing for instruction fine-tuning; Collect text data of directive language and interactive dialogue; the text data comes from online question-answering systems, chatbot logs, technical support dialogues, and other interactive platforms; the text data contains the user's specific instructions and the system's responses, showing clear instruction nature and interactive characteristics; The collected directive language and interactive dialogues are organized into a multi-round dialogue format; the collected data has two formats, namely the command response format and the dialogue format; Table 6 shows how the three corresponding formats are converted into the multi-round dialogue format, and regular expressions are used to extract and format the information; Table 6: Formatted dataset Use "You are a complete and secure conversational robot model. Complete the overall conversation content according to the user's needs, help the user complete the specified task, and pay attention to ensuring information security" as the [system instruction]; all data are converted into a unified form and then fine-tuned for instruction training; Step 32, fine-tune the instruction; Instruction fine-tuning only trains the model response part and does not calculate other inputs; for a given sequence S = system instructions: s1s2...s n User: u1u2...u m Model: m1m2...m l , only the loss of the model output part is calculated, that is, the original loss function is modified to: The modified loss function ensures that the model only learns how to reply without modeling the user's input. < / h6> < / h6>

Images