Tibetan large language model training method, electronic device and storage medium

Abstract

The embodiment of the application provides a Tibetan large language model training method, an electronic device and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: based on a Tibetan monolingual corpus set, performing continued pre-training on a preset initial Tibetan large language model to obtain a first intermediate model; based on a Tibetan-Chinese parallel corpus set, performing Tibetan-Chinese semantic correlation enhancement training on the first intermediate model to obtain a second intermediate model; based on the Tibetan-Chinese parallel corpus set, performing cross-language knowledge expansion to obtain a cross-language training sample set; based on the cross-language training sample set, performing cross-language alignment auxiliary training on the second intermediate model to obtain a third intermediate model; and based on an instruction response fine-tuning sample set, performing supervised instruction fine-tuning on the third intermediate model to obtain a target Tibetan large language model. The embodiment of the application can train the initial Tibetan large language model in stages, can reduce the gradient conflict of multi-target joint training, and can comprehensively improve the comprehensive performance of the Tibetan large language model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for training a Tibetan language large language model, an electronic device, and a storage medium. Background Technology

[0002] With the development of large language model technology, general-purpose large language models have significantly improved their text understanding, translation, question answering, and generation capabilities in high-resource languages ​​such as Chinese and English.

[0003] For Tibetan, a low-resource language, general Tibetan language models typically input Tibetan corpora directly into the model to perform autoregressive language modeling. However, existing Tibetan language models are usually trained to optimize translation quality based on single machine translation, resulting in poor performance in Tibetan knowledge-based generation tasks (such as Tibetan question answering, factual generation, domain terminology interpretation, and other comprehensive tasks). Summary of the Invention

[0004] The main objective of this application is to propose a Tibetan language large language model training method, electronic device, and storage medium, aiming to improve the performance of the Tibetan language large language model in Tibetan knowledge-based generation tasks.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for training a Tibetan large language model, the method comprising: Obtain Tibetan training parameters, which include a Tibetan monolingual corpus, a Tibetan-Chinese parallel corpus, and a command-response fine-tuning sample set; Based on the aforementioned Tibetan monolingual corpus, the preset initial Tibetan large language model is further pre-trained to obtain the first intermediate model; Based on the Tibetan-Chinese parallel corpus, the first intermediate model is trained to enhance the semantic association between Tibetan and Chinese to obtain the second intermediate model. Based on the Tibetan-Chinese parallel corpus, cross-linguistic knowledge expansion is performed to obtain a cross-linguistic training sample set, wherein at least some Tibetan-Chinese sentence pairs in the cross-linguistic training sample set are Tibetan-Chinese sentence pairs aligned with the weak corpus. Based on the cross-language training sample set, cross-language alignment-assisted training is performed on the second intermediate model to obtain the third intermediate model; Based on the instruction response fine-tuning sample set, supervised instruction fine-tuning is performed on the third intermediate model to obtain the target Tibetan language model.

[0006] To achieve the above objectives, a second aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0007] To achieve the above objectives, a third aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0008] The Tibetan language model training method, electronic device, and storage medium proposed in this application perform further pre-training on a pre-set initial Tibetan language model using a Tibetan monolingual corpus. This allows the initial Tibetan language model to adapt to Tibetan characters, grammar, and expression habits, improving its Tibetan text generation and comprehension capabilities, resulting in a first intermediate model. Then, through Tibetan-Chinese semantic association enhancement training, the first intermediate model learns bidirectional semantic mapping relationships, forming cross-language conversion capabilities, resulting in a second intermediate model. Subsequently, cross-language knowledge expansion supplements the scarce cross-language knowledge corpus in low-resource scenarios, resulting in a cross-language training sample set. The second intermediate model then undergoes cross-language alignment-assisted training, resulting in a third intermediate model. Finally, the third intermediate model is fine-tuned with supervised instructions to obtain the target Tibetan language model. Compared with related technologies, the embodiments of this application can train the initial Tibetan language model in stages. At this time, the gradient conflict of the initial Tibetan language model in multi-objective joint training is lower, and the model’s performance in various tasks such as Tibetan semantic understanding, Tibetan-Chinese bidirectional translation, Tibetan intelligent question answering, domain text generation, and factual response is comprehensively improved, resulting in a Tibetan language model with better overall performance. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the system structure for an application of the Tibetan large language model training method in one embodiment. Figure 2 A flowchart of a Tibetan large language model training method in one embodiment; Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0011] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0013] With the development of large language model technology, general-purpose large language models have significantly improved their text understanding, translation, question answering, and generation capabilities in high-resource languages ​​such as Chinese and English. However, for low-resource languages ​​such as Tibetan, general-purpose large language models typically suffer from the following problems: First, insufficient training corpora. The scale of high-quality Tibetan monolingual corpora and Tibetan-Chinese parallel corpora is limited, and the data sources are scattered, resulting in issues such as non-standard encoding, Tibetan-Chinese mixed writing, garbled characters, repeated sentences, low-information sentences, mistranslated sentences, and domain-mismatched sentences. Second, general parallel corpus cleaning methods are difficult to fully adapt to Tibetan. Existing parallel corpus cleaning methods typically use general indicators such as sentence length ratio, duplication detection, machine translation reverse verification, and sentence vector similarity, but they do not adequately consider Tibetan character structure, Tibetan syllable boundaries, Tibetan-Chinese terminology correspondence, Tibetan-Chinese syntactic differences, and the characteristics of low-resource domain texts. Third, general machine translation models mainly optimize the quality of translations from the source language to the target language, and cannot directly solve the problems of knowledge transfer, semantic representation alignment, factual generation, and question answering capability improvement in the training of Tibetan large language models. Fourth, while general-purpose multilingual large language models can handle multiple languages ​​simultaneously, their training data often focuses on high-resource languages, potentially compressing the representation space of low-resource Tibetan. This can lead to insufficient alignment between Tibetan input and its corresponding Chinese semantic representation after model encoding, thus affecting Tibetan comprehension, Tibetan-Chinese translation, Tibetan question answering, and domain terminology generation. Fifth, common pre-training methods for large language models typically involve directly inputting corpora into the model for autoregressive language modeling. This lack of fine-grained control over the quality of parallel corpora, the Tibetan-Chinese semantic correspondence, and the weights of training samples can result in mistranslations, omissions, semantic drift, and domain biases from low-quality parallel sentence pairs entering the model parameters. Therefore, a technical solution is needed for training low-resource Tibetan large language models that can perform fine-grained quality filtering on Tibetan-Chinese parallel corpora, utilize high-quality parallel sentence pairs for cross-linguistic semantic alignment, and then use the aligned representations and filtered corpora for training the Tibetan large language model.

[0014] Based on this, embodiments of this application provide a Tibetan language large language model training method, electronic device, and storage medium, aiming to improve the performance of the Tibetan language large language model in Tibetan knowledge-based generation tasks.

[0015] The Tibetan language big data model training method, electronic device and storage medium provided in this application are specifically described through the following embodiments. First, the Tibetan language big data model training method in this application embodiment is described.

[0016] The Tibetan language large-scale language model training method provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the Tibetan language large-scale language model training method, but is not limited to the above forms.

[0017] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0018] The Tibetan language large language model training method provided in this application can be applied to terminal devices or servers; however, this application will not elaborate on these applications individually. For example, as shown... Figure 1The application system shown includes a client 110 and a server device 120. The client 110 communicates with the server device 120 through a network. The server device 120 can obtain Tibetan training parameters through the client 110. The Tibetan training parameters include a Tibetan monolingual corpus, a Tibetan-Chinese parallel corpus, and a command response fine-tuning sample set. Based on the Tibetan monolingual corpus, the server device 120 performs further pre-training on a preset initial Tibetan language model to obtain a first intermediate model. Based on the Tibetan-Chinese parallel corpus, the server device 120 performs Tibetan-Chinese semantic association enhancement training on the first intermediate model to obtain a second intermediate model. Based on the Tibetan-Chinese parallel corpus, the server device 120 performs cross-language knowledge expansion to obtain a cross-language training sample set, wherein at least some Tibetan-Chinese sentence pairs in the cross-language training sample set are weakly aligned Tibetan-Chinese sentence pairs. Based on the cross-language training sample set, the server device 120 performs cross-language alignment auxiliary training on the second intermediate model to obtain a third intermediate model. Based on the command response fine-tuning sample set, the server device 120 performs supervised command fine-tuning on the third intermediate model to obtain the target Tibetan language model. The ability to train in stages reduces gradient conflicts during multi-objective joint training and comprehensively improves the model's performance across various tasks, including Tibetan semantic understanding, Tibetan-Chinese bidirectional translation, Tibetan intelligent question answering, domain text generation, and factual responses, ultimately resulting in a Tibetan language model with superior overall performance. The client 110 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server device 120 can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description of specific embodiments further illustrates this application.

[0019] Figure 2 This is an optional flowchart of the Tibetan language large language model training method provided in the embodiments of this application. Figure 2 The methods described may include, but are not limited to, the following steps.

[0020] Step S101: Obtain Tibetan training parameters, which include a Tibetan monolingual corpus, a Tibetan-Chinese parallel corpus, and a command response fine-tuning sample set. Step S102: Based on the Tibetan monolingual corpus, perform further pre-training on the preset initial Tibetan large language model to obtain the first intermediate model; Step S103: Based on the Tibetan-Chinese parallel corpus, perform Tibetan-Chinese semantic association enhancement training on the first intermediate model to obtain the second intermediate model; Step S104: Based on the Tibetan-Chinese parallel corpus, perform cross-language knowledge expansion to obtain a cross-language training sample set, wherein at least some Tibetan-Chinese sentence pairs in the cross-language training sample set are Tibetan-Chinese sentence pairs aligned with the weak corpus. Step S105: Based on the cross-language training sample set, perform cross-language alignment-assisted training on the second intermediate model to obtain the third intermediate model; Step S106: Based on the instruction response fine-tuning sample set, supervised instruction fine-tuning is performed on the third intermediate model to obtain the target Tibetan language model.

[0021] First, the initial Tibetan language model is pre-trained using a Tibetan monolingual corpus to adapt it to Tibetan characters, grammar, and expression habits, improving its text generation and comprehension capabilities, resulting in the first intermediate model. Next, through Tibetan-Chinese semantic association enhancement training, the first intermediate model learns bidirectional semantic mapping relationships, developing cross-language conversion capabilities, resulting in the second intermediate model. Then, cross-language knowledge expansion supplements the scarce cross-language knowledge corpus in low-resource scenarios, resulting in a cross-language training sample set. The second intermediate model is then subjected to cross-language alignment-assisted training, resulting in the third intermediate model. Finally, the third intermediate model undergoes supervised fine-tuning to obtain the target Tibetan language model. Compared with related technologies, the embodiments of this application can train the initial Tibetan language model in stages. At this time, the gradient conflict of the initial Tibetan language model in multi-objective joint training is lower, and the model’s performance in various tasks such as Tibetan semantic understanding, Tibetan-Chinese bidirectional translation, Tibetan intelligent question answering, domain text generation, and factual response is comprehensively improved, resulting in a Tibetan language model with better overall performance.

[0022] The Tibetan monolingual corpus refers to a collection of texts consisting solely of Tibetan language texts. It can encompass general texts, domain-specific texts, question-and-answer texts, terminology explanations, and long paragraphs, used to build the Tibetan language modeling capabilities of the model. The Tibetan-Chinese parallel corpus refers to a collection of sentence pairs consisting of semantically one-to-one correspondences between Tibetan and Chinese sentences. The instruction-response fine-tuning sample set can be a collection of supervised samples organized in the format of "instruction + input context + target response." The initial Tibetan large-scale language model can utilize open-source general-purpose large-scale language models, multilingual basic models, or self-developed basic models. The cross-lingual training sample set can be a training set composed of both native high-quality parallel corpora and Tibetan-Chinese weakly aligned corpora. Weakly aligned Tibetan-Chinese sentence pairs can be obtained by semantically expanding parallel Tibetan-Chinese sentence pairs. These weakly aligned Tibetan-Chinese sentence pairs represent a set of semantically matched but not strictly sentence-by-sentence correspondence Tibetan-Chinese sentence pairs, used to expand the scale of cross-lingual training data.

[0023] In some embodiments of this application, after obtaining three types of core data—a Tibetan monolingual corpus, a Tibetan-Chinese parallel corpus, and a command-response fine-tuning sample set—format verification and basic normalization are performed before training the corresponding model. For example, in the first stage of training, the normalized Tibetan monolingual corpus can be used as input to the initial Tibetan large language model, which is then pre-trained to adapt to the language distribution characteristics of Tibetan, outputting the trained initial Tibetan large language model as the first intermediate model. In the second stage of training, the normalized Tibetan-Chinese parallel corpus can be used as input to the first intermediate model, training the Tibetan-Chinese bidirectional semantic mapping capability based on the first intermediate model to establish surface semantic connections between the two languages, outputting the trained first intermediate model as the second intermediate model. Alternatively, the normalized command-response samples can be used as input to the third intermediate model, performing supervised fine-tuning to transform basic language capabilities into interactive task execution capabilities, outputting the trained third intermediate model as the target Tibetan large language model.

[0024] Understandably, the step of performing further pre-training on the preset initial Tibetan large language model based on the Tibetan monolingual corpus to obtain the first intermediate model includes: Step S201: Based on the sampling weights corresponding to each sample in the Tibetan monolingual corpus, the Tibetan monolingual corpus is randomly sampled to obtain Tibetan monolingual training data, wherein different sampling weights represent different quality levels. Step S202: Input the Tibetan monolingual training data into the initial Tibetan large language model, and enable the initial Tibetan large language model to perform Tibetan monolingual pre-training; the loss function of the initial Tibetan large language model is configured as autoregressive language modeling loss; Step S203: When the initial Tibetan language model fails to meet the first preset convergence condition, the process jumps to the step of randomly sampling the Tibetan monolingual corpus according to the sampling weights corresponding to each sample in the Tibetan monolingual corpus to obtain Tibetan monolingual training data. Step S204: The initial Tibetan language model that satisfies the first preset convergence condition is used as the first intermediate model.

[0025] By optimizing the Autoregressive Language Modeling Loss, the initial Tibetan language model can learn more standardized language rules from high-quality corpora during the process of adapting to Tibetan characters, word order, grammar, and discourse features. Under the premise of limited corpus size and low resource availability, it balances the amount and quality of training data, enabling the initial Tibetan language model to establish a stable and standardized basic Tibetan language modeling capability more efficiently.

[0026] The quality level is determined based on at least one dimension of the Tibetan monolingual corpus, including textual regularity, information density, domain matching, and language fluency. In some embodiments, the quality level includes three levels: high quality, medium quality, and low quality. The sampling weight indicates the probability coefficient of a sample of a corresponding quality level being selected for training; samples with higher weights appear more frequently during training.

[0027] Autoregressive language modeling loss is the core loss function for pre-training large language models. The training objective is to predict the generation probability of the next word given the preceding text. By minimizing this loss, the model learns the statistical regularities and expressive logic of language.

[0028] The first preset convergence condition is used to make the parameters of the initial Tibetan language model converge, and may include training epochs or autoregressive language modeling loss.

[0029] In some embodiments of this application, the sampling weight can be pre-configured, while in other embodiments, the sampling weight can be dynamically set in real time. The embodiments of this application will not elaborate on these aspects.

[0030] Understandably, the Tibetan-Chinese parallel corpus includes multiple parallel Tibetan-Chinese sentence pairs, each of which includes a target Tibetan sentence and a target Chinese sentence semantically matching the target Tibetan sentence; the step of training the first intermediate model to enhance Tibetan-Chinese semantic association based on the Tibetan-Chinese parallel corpus to obtain the second intermediate model includes: Step S301: Based on the target Tibetan sentence and target Chinese sentence of each parallel Tibetan-Chinese sentence pair, construct a bidirectional translation sample that corresponds one-to-one with the parallel Tibetan-Chinese sentence pair; each bidirectional translation sample includes a Chinese translation sample that semantically matches the corresponding target Tibetan sentence, and a Tibetan translation sample that semantically matches the corresponding target Chinese sentence. Step S302: Each translation sample in each bidirectional translation sample is used as an input sample and input into the first intermediate model for Tibetan-Chinese semantic association enhancement training until the first intermediate model meets the second preset convergence condition; the loss function of the first intermediate model is configured as Tibetan-Chinese bidirectional translation loss and Tibetan-Chinese domain terminology consistency constraint loss. Step S303: The first intermediate model that satisfies the second preset convergence condition is used as the second intermediate model.

[0031] Bidirectional translation samples enable bidirectional training of each parallel Tibetan-Chinese sentence pair, from Tibetan to Chinese and from Chinese to Tibetan, thus forming a closed-loop constraint for semantic mapping. Compared to one-way translation training, this approach more fully learns the semantic correspondence between Tibetan and Chinese, reduces semantic drift issues that easily occur in one-way training, and improves the accuracy of cross-language mapping. Simultaneously, consistency constraint loss can specifically constrain the translation results of domain terms, proper nouns, numbers, and other entities, addressing the issues of unstable terminology translation and numerous mistranslations and omissions in low-resource scenarios. This efficiently superimposes cross-language conversion capabilities on top of monolingual pre-training while ensuring translation quality within specialized domains.

[0032] Among them, the parallel Tibetan-Chinese sentence pairs are high-quality sentence pairs that have been screened and confirmed in the Tibetan-Chinese parallel corpus, including target Tibetan sentences and target Chinese sentences with one-to-one semantic correspondence.

[0033] During the training of the first intermediate model, Chinese translation samples can be used as input and the corresponding Tibetan translation samples as the expected output; alternatively, Tibetan translation samples can be used as input and the corresponding Chinese translation samples as the expected output.

[0034] The second preset convergence condition can be the convergence of the parameters of the first intermediate model.

[0035] In some embodiments, the Tibetan-Chinese bidirectional translation loss can be set as the main loss, and the Tibetan-Chinese domain terminology consistency constraint loss can be set as the auxiliary loss. The Tibetan-Chinese bidirectional translation loss can be calculated using the cross-entropy loss function, and the Tibetan-Chinese domain terminology consistency constraint loss can be calculated using the term matching loss function. In this case, the loss value obtained by weighting the Tibetan-Chinese bidirectional translation loss and the Tibetan-Chinese domain terminology consistency constraint loss is used to adjust the model parameters of the first intermediate model. This allows for efficient fine-tuning of parameters using low-rank adaptation (LoRA) in low-resource training scenarios to obtain a first intermediate model with converged parameters, which is also known as the second intermediate model.

[0036] Understandably, the Tibetan-Chinese parallel corpus includes multiple parallel Tibetan-Chinese sentence pairs, each of which includes a target Tibetan sentence and a target Chinese sentence semantically matching the target Tibetan sentence. The cross-linguistic knowledge expansion based on the Tibetan-Chinese parallel corpus to obtain a cross-linguistic training sample set includes: Step S401: For each target Tibetan sentence, the target Tibetan sentence is feature-encoded using a preset Tibetan-Chinese alignment encoder to obtain the target Tibetan sentence vector; Step S402: Perform semantic similarity retrieval on the target Tibetan sentence vector in the preset Tibetan-Chinese sentence vector index library to obtain Chinese knowledge fragments that semantically match the target Tibetan sentence; Step S403: Convert the Chinese knowledge fragments into Tibetan knowledge fragments, and combine each Chinese knowledge fragment with the corresponding Tibetan knowledge fragment to obtain weak corpus-aligned Tibetan-Chinese sentence pairs that correspond one-to-one with the target Tibetan sentence; Step S404: Based on the weak corpora described above, align Tibetan-Chinese sentence pairs and parallel Tibetan-Chinese sentence pairs to construct a cross-language training sample set.

[0037] First, a Tibetan-Chinese alignment encoder and sentence vector index library, pre-trained through contrastive learning, are used to retrieve Chinese knowledge fragments by semantic retrieval using Tibetan sentences as queries. These fragments are then converted into Tibetan-Chinese weakly aligned corpora, ensuring a high semantic relevance between the expanded corpora and the original sentences. This method has lower noise compared to keyword matching or randomly generated expansion methods, alleviating the pain point of scarce Tibetan language corpora in low-resource domains. At the same time, the weakly aligned corpora and the original high-quality parallel corpora together form a cross-language training sample set, which supplements the domain knowledge coverage without compromising translation accuracy, providing more sufficient data support for subsequent alignment-assisted training.

[0038] The Tibetan-Chinese alignment encoder can be an encoder trained through cross-language contrastive learning. For example, using high-quality sentence pairs as positive samples and multi-class difficult negative samples as negative samples, the Tibetan-Chinese alignment encoder can be trained using Information Noise Contrast Estimation (InfoNCE) loss. The Tibetan-Chinese alignment encoder can map Tibetan and Chinese sentences to the same semantic vector space.

[0039] The Tibetan-Chinese sentence vector index is a pre-built, large-scale Chinese sentence vector retrieval database. It stores a vast amount of Chinese knowledge fragments and their corresponding sentence vectors, supporting fast semantic retrieval based on vector similarity. Chinese knowledge fragments are text segments retrieved from the index that highly match the semantics of the target Tibetan sentence. Tibetan knowledge fragments are text segments obtained by converting Chinese knowledge fragments into Tibetan, establishing a semantic correspondence with the Chinese knowledge fragments.

[0040] Weakly aligned Tibetan-Chinese sentence pairs are sets of sentence pairs composed of Chinese knowledge fragments and corresponding Tibetan knowledge fragments. Compared with strictly manually annotated parallel corpora, they represent overall semantic matching rather than strict word-by-word correspondence.

[0041] Understandably, the cross-language training sample set includes pairs of Tibetan-Chinese sentences, each pair comprising a corresponding Tibetan sentence and a Chinese sentence. Based on this cross-language training sample set, cross-language alignment-assisted training is performed on the second intermediate model to obtain a third intermediate model, which includes: Step 501: For each sample Tibetan-Chinese sentence pair, the sample Tibetan sentence and sample Chinese sentence in the sample Tibetan-Chinese sentence pair are feature encoded by a preset Tibetan-Chinese alignment encoder to obtain the first Tibetan sentence vector and the first Chinese sentence vector. Step 502: Calculate the first cosine similarity between the first Tibetan sentence vector and the corresponding first Chinese sentence vector, and use the first cosine similarity as the sample training weight of the sample Tibetan-Chinese sentence pair. Step 503: Input each of the sample Tibetan-Chinese sentence pairs into the second intermediate model, so that the second intermediate model performs cross-language alignment-assisted training until the second intermediate model meets the third preset convergence condition; the loss function of the second intermediate model is calculated based on the sample training weights and the cross-language representation consistency constraint loss function. Step 504: The second intermediate model that satisfies the third preset convergence condition is used as the third intermediate model.

[0042] The training weights of each Tibetan-Chinese sentence pair are assigned based on the cosine similarity of the Tibetan-Chinese sentence vectors. At the same time, cross-language representation consistency constraints are added. Then, the Tibetan-Chinese semantic space is forcibly aligned from the middle layer of the model through representation consistency constraints. Compared with the surface generation alignment of translation tasks alone, this alignment at the representation level can achieve a deeper semantic space unification, allowing Chinese knowledge to be transferred to Tibetan representations more directly and efficiently, and further deepening the degree of Tibetan-Chinese semantic alignment.

[0043] A Tibetan-Chinese sentence pair refers to each corresponding Tibetan-Chinese sentence pair in the cross-language training sample set, containing a sample Tibetan sentence and a sample Chinese sentence. The first Tibetan sentence vector and the first Chinese sentence vector are the semantic vectors obtained after encoding the Tibetan and Chinese sentences in the sample Tibetan-Chinese sentence pair by the Tibetan-Chinese alignment encoder. The first cosine similarity is the cosine similarity value between the Tibetan sentence vector and the Chinese sentence vector in the same sample sentence pair, and its value can be in the range of [-1, 1]. The higher the value, the higher the semantic alignment. The sample training weight refers to the weight coefficient of each sample in the training loss; the higher the similarity, the greater the weight.

[0044] Cross-language representation consistency constraint loss can be cosine similarity loss, KL divergence loss, etc., with the training objective being to make the semantically equivalent Tibetan and Chinese inputs output similar representation vectors in the intermediate semantic layer of the model, thus aligning the semantic spaces of the two languages. The third preset convergence condition can be the convergence of the parameters of the second intermediate model.

[0045] Understandably, the process of performing supervised instruction fine-tuning on the third intermediate model based on the instruction response fine-tuning sample set to obtain the target Tibetan large language model includes: Step S601: Normalize the format and structure the fields of each sample in the instruction response fine-tuning sample set to obtain structured instruction response samples. The instruction response fine-tuning sample set includes at least one of the following: Tibetan semantic understanding response samples, Tibetan-Chinese bidirectional translation response samples, Tibetan text summary response samples, Tibetan text classification response samples, and Tibetan terminology interpretation response samples. Step S602: Input the instruction field and input context field in the structured instruction response sample into the third intermediate model, and use the target response field corresponding to the structured instruction response sample as the supervision label to perform supervised instruction fine-tuning training on the third intermediate model until the third intermediate model meets the fourth preset convergence condition. Step S603: The third intermediate model that satisfies the fourth preset convergence condition is taken as the target Tibetan language model.

[0046] By unifying instruction samples from multiple sources into a standardized structure through format normalization, the unified input-output paradigm helps the model quickly grasp the rules of instruction adherence, improves its generalization ability to different task instructions, and avoids the instability of instruction adherence caused by chaotic sample formats. Secondly, it covers samples from multiple tasks such as semantic understanding, bidirectional translation, text summarization, text classification, and terminology definition, which can simultaneously enhance the model's comprehensive task capabilities and promote knowledge transfer between different tasks. This is more efficient than training by fine-tuning a single task. Furthermore, placing instruction fine-tuning after training all basic capabilities can efficiently transform underlying capabilities into directly applicable task execution capabilities without compromising underlying language modeling and cross-language capabilities, thus adapting to more downstream application scenarios.

[0047] The structured instruction response samples are training samples that have undergone format normalization and are uniformly structured into a three-part structure of "instruction field + input context field + target response field". They serve as the standard input unit for supervised instruction fine-tuning (SFT).

[0048] The Tibetan semantic understanding response samples are instruction samples for Tibetan text classification, entity recognition, semantic matching, and stance recognition tasks, used to train the model's Tibetan semantic parsing capabilities. The Tibetan-Chinese bidirectional translation response samples are instruction samples for Tibetan-to-Chinese and Chinese-to-Tibetan translation tasks, used to solidify and optimize the model's cross-language translation capabilities.

[0049] The Tibetan text summarization response samples are instruction samples for tasks involving summarizing and extracting key points from long Tibetan texts, used to train the model's ability to compress and generate Tibetan information. The Tibetan text classification response samples are instruction samples for tasks involving classification, sentiment classification, and topic classification within the Tibetan language domain.

[0050] The Tibetan terminology definition response samples are instruction samples for tasks involving the explanation of Tibetan-language terms and Tibetan-Chinese terminology comparison, used to enhance the model's terminology processing capabilities. The instruction field is a text field describing the task requirements, clarifying the specific tasks the model needs to perform. The input context field is the object text for the task processing, i.e., the original Tibetan / Chinese text that the model needs to understand, translate, summarize, etc. The target response field is the standard answer text corresponding to the task, serving as a label for supervised training.

[0051] Understandably, the Tibetan-Chinese parallel corpus is obtained through the following steps: Step S701: Obtain the original Tibetan-Chinese parallel corpus; Step S702: Preprocess the original Tibetan-Chinese parallel corpus to obtain a candidate sentence pair set. The preprocessing includes at least one of Tibetan character normalization, Tibetan word segmentation and sub-word segmentation, sentence pair length ratio detection, and garbled text and duplicate sentence filtering. Step S703: For each candidate Tibetan-Chinese sentence in the candidate Tibetan-Chinese sentence pair set and the candidate Chinese sentence in the candidate Tibetan-Chinese sentence pair set, a target vocabulary coverage score is performed to obtain the vocabulary coverage score corresponding to the candidate Tibetan-Chinese sentence pair. Step S704: Translate candidate Tibetan sentences into target Chinese and candidate Chinese sentences into target Tibetan using a preset Tibetan-Chinese translation model; obtain the first semantic similarity between the target Tibetan and the corresponding candidate Tibetan sentences, and the second semantic similarity between the target Chinese and the corresponding candidate Chinese sentences; obtain a bidirectional translation consistency score corresponding to the candidate Tibetan-Chinese sentence pair based on the first semantic similarity and the second semantic similarity of the same candidate Tibetan-Chinese sentence pair. Step S705: Determine the Tibetan-Chinese parallel corpus based on the vocabulary coverage score and the bidirectional translation consistency score.

[0052] By employing Tibetan-specific preprocessing operations to filter out obviously invalid samples such as garbled characters, repetitions, abnormal lengths, and language mismatches, most format-related noise is quickly eliminated, reducing the computational burden of subsequent scoring. Operations such as Tibetan character normalization, Tibetan word segmentation, and syllable validity verification are adapted to the character structure and syllable characteristics of Tibetan. The quality of sentence pairs is then evaluated from two dimensions: lexical coverage and bidirectional translation consistency. Lexical coverage filters mismatched samples from the surface matching dimension of terms, proper nouns, and entities, while bidirectional translation consistency identifies mistranslated or omitted sentence pairs from the deep dimension of semantic restoration. Furthermore, the bidirectional back-translation method avoids the bias of a single translation model, enabling more accurate selection of semantically corresponding parallel sentence pairs and improving the overall quality of the training corpus from the data source.

[0053] The original Tibetan-Chinese parallel corpus can be an initial, unwashed set of Tibetan-Chinese sentence pairs collected from multiple sources, including public corpora, document alignment extraction, web scraping, and machine translation generation. It commonly suffers from format noise, semantic mismatches, and domain bias. The candidate sentence pair set refers to the set of sentence pairs from the original corpus after preliminary preprocessing to remove obviously invalid samples. A candidate Tibetan-Chinese sentence pair is a single sentence pair in the candidate sentence pair set, containing both a candidate Tibetan sentence and a candidate Chinese sentence. The vocabulary coverage score is based on a Tibetan-Chinese dictionary, a domain terminology list, and a proper noun lookup table. It quantifies the correspondence of sentence pairs from the dimensions of vocabulary, terminology, and entity matching; a higher score indicates a better correspondence between vocabulary and terminology. The Tibetan-Chinese translation model is a benchmark model with bidirectional Tibetan-Chinese translation capabilities, used to perform bidirectional translation consistency detection.

[0054] The target Chinese text refers to the Chinese translation generated after inputting candidate Tibetan sentences into the Tibetan-Chinese translation model. The target Tibetan text refers to the Tibetan translation generated after inputting candidate Chinese sentences into the Tibetan-Chinese translation model.

[0055] The first semantic similarity score refers to the numerical value of semantic similarity between the target Tibetan sentence obtained through back-translation and the original candidate Tibetan sentence. The second semantic similarity score refers to the numerical value of semantic similarity between the target Chinese sentence obtained through back-translation and the original candidate Chinese sentence. The bidirectional translation consistency score is a score obtained by combining the first and second semantic similarities, used to measure the bidirectional semantic restoration ability of the sentence pair. The higher the score, the more reliable the semantic correspondence of the sentence pair.

[0056] In this embodiment, format unification and basic deduplication can be performed on the original Tibetan-Chinese parallel corpus from multiple sources. Then, preprocessing is performed on the original Tibetan-Chinese parallel corpus, including at least one of the following operations: Tibetan character normalization, Tibetan word segmentation and sub-word segmentation, sentence pair length ratio detection, garbled text and duplicate sentence filtering, Tibetan-Chinese language identification, and special symbol and proper noun normalization. After removing obviously invalid samples, a candidate sentence pair set is obtained. Subsequently, for each candidate Tibetan-Chinese sentence pair, based on a Tibetan-Chinese dictionary, a domain terminology table, and a proper noun comparison table, the terms, proper nouns, and numbers on both sides of the Tibetan-Chinese language are statistically analyzed. The matching of characters and entities is used to calculate the lexical coverage score of the sentence pair. Then, a preset Tibetan-Chinese translation model can be called to translate the candidate Tibetan sentences into the target Chinese and the candidate Chinese sentences into the target Tibetan. The first semantic similarity between the target Tibetan and the original Tibetan, and the second semantic similarity between the target Chinese and the original Chinese are calculated respectively. Based on the similarity in both directions, the bidirectional translation consistency score of the sentence pair is obtained. Finally, the lexical coverage score and the bidirectional translation consistency score can be combined to select the sentence pairs that meet the score, which are then used as a Tibetan-Chinese parallel corpus.

[0057] Understandably, determining the Tibetan-Chinese parallel corpus based on the vocabulary coverage score and the bidirectional translation consistency score includes: Step S801: Encode the candidate Tibetan sentence and the candidate Chinese sentence respectively using a preset multilingual sentence vector encoder to obtain the second Tibetan sentence vector and the second Chinese sentence vector; obtain the second cosine similarity between the second Tibetan sentence vector and the second Chinese sentence vector; and obtain the cross-language sentence vector matching score corresponding to the candidate Tibetan-Chinese sentence pair based on the second cosine similarity. Step S802: The candidate Tibetan sentences and candidate Chinese sentences are classified into different domains using a pre-trained domain classifier to obtain domain classification results. Scoring is performed based on the domain classification results to obtain a domain relevance score corresponding to the candidate Tibetan-Chinese sentence pair. Step S803: Calculate the first perplexity of the candidate Tibetan sentence using a preset Tibetan language perplexity model, calculate the second perplexity of the candidate Chinese sentence using a preset Chinese language perplexity model, and obtain the language fluency score corresponding to the candidate Tibetan-Chinese sentence pair based on the first perplexity and the corresponding second perplexity. Step S804: The comprehensive quality score for each candidate Tibetan-Chinese sentence pair is obtained by weighted summation of the vocabulary coverage score, bidirectional translation consistency score, cross-language sentence vector matching score, domain relevance score and language fluency score. Step S805: Select Tibetan-Chinese sentence pairs with a comprehensive quality score higher than a preset threshold as the Tibetan-Chinese parallel corpus set.

[0058] Building upon lexical coverage and consistency in bidirectional translation, this paper supplements three scoring dimensions: cross-language sentence vector matching, domain relevance, and language fluency. This forms a multi-dimensional scoring system covering lexical surface, vector semantics, domain adaptation, and language naturalness. Each dimension characterizes sentence pair quality from different perspectives, enabling the identification of noise samples that cannot be detected by a single dimension, thus significantly improving the accuracy of quality evaluation. The scores from each dimension are integrated through weighted summation, and the weights can be adjusted to adapt to different application scenarios, such as general and domain-specific ones, making it more flexible. Finally, the overall quality score is used to classify the grades, providing a guarantee for the effectiveness of subsequent model training.

[0059] The multilingual sentence vector encoder is a pre-trained sentence vector model supporting both Tibetan and Chinese languages. It converts text into fixed-dimensional semantic vectors for calculating cross-language semantic similarity. The second Tibetan sentence vector and second Chinese sentence vector refer to the semantic vectors obtained after encoding candidate Tibetan and candidate Chinese sentences by the multilingual sentence vector encoder. The second cosine similarity is the cosine similarity between the second Tibetan sentence vector and the second Chinese sentence vector, used to measure the degree of cross-language semantic matching. The cross-language sentence vector matching score is based on the second cosine similarity transformation; a higher score indicates a better cross-language semantic matching. The pre-trained domain classifier is a pre-trained text domain classification model that can identify the category of the text, including general, government affairs, news, laws and regulations, and ethnic culture. The domain classification result is the domain label and confidence score output by the domain classifier for Tibetan and Chinese sentences, respectively. The domain relevance score is calculated based on the consistency of the domain classifications on both sides of the Tibetan and Chinese sides and the matching degree with the target application domain; a higher score indicates a better domain matching degree for the sentence pair. The Tibetan language perplexity model is a specialized Tibetan language model used to calculate the perplexity of Tibetan sentences, measuring their naturalness and fluency. The first perplexity level represents the perplexity score of candidate Tibetan sentences; a higher perplexity score indicates a more unnatural sentence and a greater likelihood of grammatical errors or noise. The Chinese language perplexity model is a specialized Chinese language model used to calculate the perplexity of Chinese sentences. The second perplexity level represents the perplexity score of candidate Chinese sentences.

[0060] The language fluency score is a positive score derived from the conversion between the first and second perplexity levels; a higher score indicates better naturalness of the sentence. The overall quality score is the weighted sum of scores from multiple dimensions, serving as the final measure of overall sentence quality. Weight parameters are the proportion of each scoring dimension in the overall score and can be adjusted and optimized based on manually labeled validation sets to suit different application needs. Preset thresholds are the critical score values ​​used to classify sample quality levels, including high-quality and low-quality thresholds, corresponding to classifying samples into high-quality, questionable, and low-quality levels.

[0061] In this embodiment, a pre-set multilingual sentence vector encoder can be invoked to encode the candidate Tibetan and candidate Chinese sentences respectively, obtaining the second Tibetan sentence vector and the second Chinese sentence vector. The second cosine similarity between the two is calculated, and the cross-language sentence vector matching score of this sentence pair is obtained. Next, a pre-trained domain classifier can be invoked to classify the candidate Tibetan and candidate Chinese sentences into their respective domains, obtaining the domain classification results on both sides. Combining the domain label consistency and the matching degree with the target domain, the domain relevance score of this sentence pair is calculated. Subsequently, the Tibetan and Chinese language perplexity models are invoked respectively to calculate the first perplexity of the candidate Tibetan sentence and the second perplexity of the candidate Chinese sentence. The perplexity score is converted into a positive language fluency score. Then, vocabulary coverage score, bidirectional translation consistency score, cross-language sentence vector matching score, domain relevance score, and language fluency score are combined with auxiliary features such as length ratio score and language confidence score, and weighted summation according to preset weight parameters to obtain the comprehensive quality score corresponding to each candidate Tibetan-Chinese sentence pair. Finally, based on the comprehensive quality score and preset threshold, the samples are divided into three levels: high quality, questionable, and low quality. Samples with a comprehensive quality score higher than the high quality threshold are included in the Tibetan-Chinese parallel corpus. Questionable samples can be dynamically determined whether to be included with reduced weight according to the training stage, and low quality samples are directly removed from the training set.

[0062] As an example, the process of determining the Tibetan-Chinese parallel corpus is as follows: Step 1: First, obtain the original Tibetan-Chinese parallel corpus. : Original Tibetan-Chinese Parallel Corpus Each sample in the dataset can be represented as: sample ,in, Representing Tibetan sentences, This indicates the corresponding Chinese sentence. This represents sample metadata. Metadata may include information such as data source, collection time, domain tags, document number, paragraph number, whether it was manually annotated, whether it was generated by machine translation, whether it came from a public corpus, and whether it came from a domain terminology database.

[0063] Step 2: Standardization of Tibetan characters: In some embodiments, Tibetan sentences Preprocessing for character normalization includes: This process converts Tibetan characters from different encoding sources to Unicode, also known as Unified Code encoding. It standardizes Tibetan syllable separators and unifies the processing of the Tibetan tsheg symbol. It unifies full-width and half-width characters, removes invisible control characters, abnormal spaces, and meaningless duplicate symbols, and standardizes the mapping of Tibetan, Chinese, and mixed punctuation. It establishes mapping tables for common variant characters, incorrectly encoded characters, or characters with similar display characteristics, replacing them without altering the semantic meaning. It preserves semantically meaningful numbers, dates, place names, personal names, organization names, and technical terms. The output is a standardized Tibetan sentence. .

[0064] Step 3: Tibetan word segmentation or sub-word splitting: In some embodiments, the standardized Tibetan sentences Preprocessing is performed for word segmentation or sub-word segmentation. Specific methods may include: initial syllable segmentation based on Tibetan syllable delimiters; longest-match segmentation based on a Tibetan dictionary and domain terminology; sub-word segmentation methods for out-of-vocabulary words, such as byte-pair encoding or single-character / syllable-level segmentation; preservation of proper names, numbers, timestamps, units, and abbreviations; and validation of the segmentation results, marking consecutive abnormal symbols, excessively long unknown segments, and abnormally repeated syllables as low-quality candidates. The output is a Tibetan segmentation sequence. For Chinese sentences It can perform Chinese word segmentation, punctuation normalization, number normalization, and proper name recognition, and output Chinese character segmentation sequences. .

[0065] Step 4: Sentence Pair Length Ratio Detection: In some embodiments, the length features of Tibetan and Chinese sentences are calculated. Length features may include: the number of Tibetan characters; the number of Tibetan syllables; the number of Tibetan subwords; the number of Chinese characters; the number of Chinese words; and the number of digits, punctuation marks, and proper nouns. The normal range for the Tibetan-Chinese length ratio is obtained statistically based on the training corpus. For example, the ratio distribution of Tibetan syllables to Chinese words can be statistically analyzed on manually verified high-quality seed corpora to determine upper and lower thresholds. If the length ratio of a sentence pair significantly deviates from the normal range, it is marked as suspicious or low-quality. This step does not directly delete all sentence pairs with abnormal lengths; instead, it uses abnormal length ratios as a feature in the quality score to avoid mistakenly deleting special but useful samples such as short headings, glossaries, and question-and-answer corpora.

[0066] Step 5: Filtering out garbled text, repetitive sentences, and low-information sentences: In some embodiments, the following filtering is performed on each sentence pair: detecting the proportion of unrecognizable characters, the proportion of abnormal coded segments, the proportion of control characters, and the proportion of repeated meaningless symbols; calculating hash values ​​for Tibetan sentences, Chinese sentences, and Tibetan-Chinese sentence pairs as a whole to remove completely duplicate samples; and using character... - Deduplication is performed using meta-fragment similarity or sentence vector similarity; sentences containing only punctuation, numbers, URLs, templated phrases, or words without substantial semantic meaning are identified; samples with empty Tibetan or Chinese text, or with a valid character length below a preset threshold, are deleted; the proportion of consecutive repeated syllables, repeated characters, or repeated phrases is detected, and abnormal proportions are marked as low quality. The output is a set of candidate sentence pairs after preliminary filtering. .

[0067] Step Six: Identification of Tibetan and Chinese Language: In some embodiments, for Each sentence pair in the code undergoes language identification. Language identification can employ a combination of rules and models: determining the proportion of Tibetan, Chinese, Latin, and other characters based on the Unicode character range; determining the legality of Tibetan segments based on Tibetan syllable structure; using a trained language identification classifier to determine the language at the sentence level; for sentences containing mixed Tibetan and Chinese text, labeling the proportion and location of the mixed text; and removing or correcting samples where the Tibetan column is actually Chinese, the Chinese column is actually Tibetan, both columns are the same language, or both columns are garbled characters. The output includes language labels. and language confidence .

[0068] Step 7: Standardization of special symbols, numbers, punctuation, and proper names: In some embodiments, special symbols, numbers, punctuation, and proper names in sentence pairs are standardized, including: mapping Arabic numerals, Tibetan numerals, and Chinese numerals to a unified representation, or preserving the original form and establishing alignment tags; standardizing dates, times, amounts, percentages, units of measurement, etc., into structured tags; identifying personal names, place names, organization names, policy names, and domain terms; using Tibetan-Chinese terminology lists or proper name lookup tables to mark correspondences; and performing consistency checks on missing numbers, dates, or proper names in both sentences that should correspond. The output is a set of preprocessed candidate sentence pairs. .

[0069] Step 8: Sentence quality assessment: In some embodiments, for the candidate sentence pair set Each sentence in the text is used to calculate multiple quality scores.

[0070] (1) Vocabulary Coverage Scoring. Based on Tibetan-Chinese dictionaries, domain terminology lists, and proper noun comparison tables, the coverage of keywords, terms, or proper nouns in Tibetan sentences in Chinese sentences, and the correspondence of domain terms in Chinese sentences in Tibetan sentences, are calculated. The vocabulary coverage score can be calculated based on the following factors: the number of Tibetan-Chinese term matches; the number of missing terms; the number of identical proper nouns; the number of identical numbers, dates, and units; and the coverage ratio of domain keywords. Output the vocabulary coverage score. .

[0071] (2) Bidirectional translation consistency scoring. Using existing Tibetan-Chinese translation models or initial translation models to be trained, bidirectional translation consistency testing is performed on sentence pairs: Tibetan sentences are compared with Chinese sentences. Candidates for translation into Chinese ; Translate Chinese sentences Translated into Tibetan candidate ;Compare With the original Chinese Semantic similarity; comparison With the original Tibetan script Semantic similarity; checks whether numbers, proper names, and terms remain consistent in bidirectional translation. Outputs a bidirectional translation consistency score. This step does not require the translation model itself to achieve high performance; it is only used as a semantic consistency detection tool. For samples with low confidence levels in the translation model, the weight of this score in the overall score can be reduced.

[0072] (3) Cross-language sentence vector matching and scoring. Using a multilingual sentence vector model and a Tibetan-Chinese pre-trained encoder, Tibetan sentences are matched and scored. Chinese sentences Encode into vectors respectively and Calculate the cosine similarity between the two to obtain the cross-language matching score. If a Tibetan sentence in the same batch has a higher similarity to a non-corresponding Chinese sentence than its corresponding Chinese sentence, it is marked as suspicious.

[0073] (4) Domain Relevance Scoring. Based on pre-defined domain labels for the project or application scenario, such as general text, government documents, news articles, legal regulations, ethnic language and culture texts, public opinion analysis texts, and information security texts, a domain classifier is used to determine whether sentence pairs belong to the target domain. Domain Relevance Scoring The following factors can be used to determine the domain classification results: Tibetan sentence domain classification results; Chinese sentence domain classification results; whether the domain labels on both sides are consistent; whether the sentence pair contains target domain terms; and whether the source metadata of the sentence pair is reliable.

[0074] (5) Language fluency scoring. Sentence perplexity was calculated using both Tibetan and Chinese language models. Perplexity measures how natural a sentence appears to the language model. Sentences with abnormally high perplexity may contain grammatical errors, garbled text, machine translation noise, or unnatural text. Tibetan and Chinese perplexity scores were then converted into language fluency scores. To avoid deleting text dense with technical terms, the perplexity threshold can be adjusted in conjunction with domain tags.

[0075] (6) Integration of overall quality scores. The above scores are integrated into a sentence-to-sentence quality score. The fusion method can be weighted summation, a learned ranking model, or a classification model. One implementation of weighted summation is as follows:

[0076] in, Indicates the length ratio score. The confidence score for language identification is represented. to These are the weights corresponding to different types of ratings. The weights can be adjusted based on manually labeled validation sets. The length ratio score is obtained by comparing the lengths of candidate Tibetan sentences with those of candidate Chinese sentences. The score is obtained based on the language identification results, which are obtained by performing language identification on candidate Tibetan sentences and candidate Chinese sentences respectively through a language identification model. The language identification results include the confidence scores of candidate Tibetan sentences and candidate Chinese sentences.

[0077] Step Nine: Dynamic Filtering and Sample Reweighting In some embodiments, based on the overall quality score Sentence pairs are divided into three levels. Above the first threshold The sentence pairs are high-quality samples, which are the Tibetan-Chinese parallel corpus; Between and Samples between these thresholds are considered suspicious. These samples can be added to a reduced-weight training set or placed in a manual review queue. Below the second threshold The samples are low-quality samples, and these samples are removed from the training set.

[0078] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described Tibetan language large-scale language model training method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0079] Please see Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the Tibetan language large language model training method of the embodiments of this application. The 903 input / output interface is used to implement information input and output. The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0080] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described Tibetan language large language model training method.

[0081] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0082] The Tibetan language model training method, electronic device, and storage medium provided in this application embodiment perform further pre-training on a preset initial Tibetan language model using a Tibetan monolingual corpus. This allows the initial Tibetan language model to adapt to Tibetan characters, grammar, and expression habits, solidifying the underlying foundation for Tibetan text generation and understanding, resulting in a first intermediate model. Then, through Tibetan-Chinese semantic association enhancement training, the first intermediate model learns bidirectional semantic mapping relationships, forming cross-language conversion capabilities, resulting in a second intermediate model. Subsequently, cross-language knowledge expansion supplements the scarce cross-language knowledge corpus in low-resource scenarios, resulting in a cross-language training sample set. Next, cross-language alignment-assisted training is performed on the second intermediate model to obtain a third intermediate model. Finally, supervised fine-tuning of the third intermediate model yields the target Tibetan language model. The ability to train the initial Tibetan language model in stages can reduce gradient conflicts in multi-objective joint training and comprehensively improve the model's performance on various tasks such as Tibetan semantic understanding, Tibetan-Chinese bidirectional translation, Tibetan intelligent question answering, domain text generation, and factual responses, ultimately resulting in a Tibetan language model with better overall performance.

[0083] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0084] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0085] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0086] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0087] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0088] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0090] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for training a large Tibetan language model, characterized in that, The method includes: Obtain Tibetan training parameters, which include a Tibetan monolingual corpus, a Tibetan-Chinese parallel corpus, and a command-response fine-tuning sample set; Based on the aforementioned Tibetan monolingual corpus, the preset initial Tibetan large language model is further pre-trained to obtain the first intermediate model; Based on the Tibetan-Chinese parallel corpus, the first intermediate model is trained to enhance the semantic association between Tibetan and Chinese to obtain the second intermediate model. Based on the Tibetan-Chinese parallel corpus, cross-linguistic knowledge expansion is performed to obtain a cross-linguistic training sample set, wherein at least some Tibetan-Chinese sentence pairs in the cross-linguistic training sample set are weakly aligned Tibetan-Chinese sentence pairs; Based on the cross-language training sample set, cross-language alignment-assisted training is performed on the second intermediate model to obtain the third intermediate model; Based on the instruction response fine-tuning sample set, supervised instruction fine-tuning is performed on the third intermediate model to obtain the target Tibetan language model.

2. The method according to claim 1, characterized in that, The step of performing further pre-training on the preset initial Tibetan language model based on the Tibetan monolingual corpus to obtain a first intermediate model includes: Based on the sampling weights corresponding to each sample in the Tibetan monolingual corpus, random sampling is performed on the Tibetan monolingual corpus to obtain Tibetan monolingual training data, wherein different sampling weights represent different quality levels; The Tibetan monolingual training data is input into the initial Tibetan large language model, and the initial Tibetan large language model is then used to perform further pre-training of Tibetan monolingual data; the loss function of the initial Tibetan large language model is configured as an autoregressive language modeling loss. When the initial Tibetan language model fails to meet the first preset convergence condition, the process jumps to the step of randomly sampling the Tibetan monolingual corpus according to the sampling weights corresponding to each sample in the Tibetan monolingual corpus to obtain Tibetan monolingual training data. The initial Tibetan language model that meets the first preset convergence condition is used as the first intermediate model.

3. The method according to claim 1, characterized in that, The Tibetan-Chinese parallel corpus includes multiple parallel Tibetan-Chinese sentence pairs, each of which includes a target Tibetan sentence and a target Chinese sentence that semantically matches the target Tibetan sentence. The second intermediate model is obtained by performing Tibetan-Chinese semantic association enhancement training on the first intermediate model based on the Tibetan-Chinese parallel corpus, including: Based on the target Tibetan sentence and target Chinese sentence for each of the parallel Tibetan-Chinese sentence pairs, a bidirectional translation sample corresponding one-to-one with the parallel Tibetan-Chinese sentence pairs is constructed; each bidirectional translation sample includes a Chinese translation sample that semantically matches the corresponding target Tibetan sentence, and a Tibetan translation sample that semantically matches the corresponding target Chinese sentence. Each translation sample in each of the bidirectional translation samples is used as an input sample and input into the first intermediate model for Tibetan-Chinese semantic association enhancement training until the first intermediate model satisfies the second preset convergence condition; the loss function of the first intermediate model is configured as Tibetan-Chinese bidirectional translation loss and Tibetan-Chinese domain terminology consistency constraint loss; The first intermediate model that satisfies the second preset convergence condition is used as the second intermediate model.

4. The method according to claim 1, characterized in that, The Tibetan-Chinese parallel corpus includes multiple parallel Tibetan-Chinese sentence pairs. Each parallel Tibetan-Chinese sentence pair includes a target Tibetan sentence and a target Chinese sentence that semantically matches the target Tibetan sentence. The cross-linguistic knowledge expansion based on the Tibetan-Chinese parallel corpus yields a cross-linguistic training sample set, including: For each target Tibetan sentence, the target Tibetan sentence is feature-encoded using a preset Tibetan-Chinese aligned encoder to obtain the target Tibetan sentence vector; Semantic similarity retrieval is performed on the target Tibetan sentence vector in the preset Tibetan-Chinese sentence vector index library to obtain Chinese knowledge fragments that semantically match the target Tibetan sentence; The Chinese knowledge fragments are converted into Tibetan knowledge fragments, and each Chinese knowledge fragment is combined with the corresponding Tibetan knowledge fragment to obtain weak corpus-aligned Tibetan-Chinese sentence pairs that correspond one-to-one with the target Tibetan sentence. Based on the weak corpora described, Tibetan-Chinese sentence pairs and parallel Tibetan-Chinese sentence pairs are aligned to construct a cross-language training sample set.

5. The method according to claim 1 or 4, characterized in that, The process of performing cross-language alignment-assisted training on the second intermediate model based on the cross-language training sample set to obtain a third intermediate model includes: For each sample Tibetan-Chinese sentence pair, the sample Tibetan sentence and sample Chinese sentence in the sample Tibetan-Chinese sentence pair are respectively feature-encoded by a preset Tibetan-Chinese alignment encoder to obtain the first Tibetan sentence vector and the first Chinese sentence vector; Calculate the first cosine similarity between the first Tibetan sentence vector and the corresponding first Chinese sentence vector, and use the first cosine similarity as the sample training weight of the sample Tibetan-Chinese sentence pair; Each of the sample Tibetan-Chinese sentence pairs is input into the second intermediate model, and the second intermediate model performs cross-language alignment-assisted training until the second intermediate model meets the third preset convergence condition; the loss function of the second intermediate model is calculated based on the sample training weights and the cross-language representation consistency constraint loss function; The second intermediate model that satisfies the third preset convergence condition is used as the third intermediate model.

6. The method according to claim 1, characterized in that, The process of performing supervised instruction fine-tuning on the third intermediate model based on the instruction response fine-tuning sample set to obtain the target Tibetan language model includes: Each sample in the instruction response fine-tuning sample set is subjected to format normalization and field structuring to obtain structured instruction response samples. The instruction response fine-tuning sample set includes at least one of the following: Tibetan semantic understanding response samples, Tibetan-Chinese bidirectional translation response samples, Tibetan text summary response samples, Tibetan text classification response samples, and Tibetan terminology interpretation response samples. The instruction field and input context field in the structured instruction response sample are input into the third intermediate model. The target response field corresponding to the structured instruction response sample is used as the supervision label to perform supervised instruction fine-tuning training on the third intermediate model until the third intermediate model meets the fourth preset convergence condition. The third intermediate model that satisfies the fourth preset convergence condition is taken as the target Tibetan language model.

7. The method according to any one of claims 1-6, characterized in that, The Tibetan-Chinese parallel corpus was obtained through the following steps: Obtain the original Tibetan-Chinese parallel corpus; The original Tibetan-Chinese parallel corpus is preprocessed to obtain a candidate sentence pair set. The preprocessing includes at least one of Tibetan character normalization, Tibetan word segmentation and sub-word segmentation, sentence pair length ratio detection, and garbled text and duplicate sentence filtering. For each candidate Tibetan-Chinese sentence pair in the candidate sentence pair set, the candidate Tibetan sentence and the candidate Chinese sentence are subjected to target vocabulary coverage scoring to obtain the vocabulary coverage score corresponding to the candidate Tibetan-Chinese sentence pair; The candidate Tibetan sentences are translated into the target Chinese sentences and the candidate Chinese sentences are translated into the target Tibetan sentences using a pre-defined Tibetan-Chinese translation model. Obtain the first semantic similarity between the target Tibetan text and the corresponding candidate Tibetan text sentences, and the second semantic similarity between the target Chinese text and the corresponding candidate Chinese text sentences; Based on the first semantic similarity and the second semantic similarity of the same candidate Tibetan-Chinese sentence pair, a bidirectional translation consistency score corresponding to the candidate Tibetan-Chinese sentence pair is obtained; Based on the vocabulary coverage score and the bidirectional translation consistency score, a Tibetan-Chinese parallel corpus was determined.

8. The method according to claim 7, characterized in that, The process of determining the Tibetan-Chinese parallel corpus based on the vocabulary coverage score and the bidirectional translation consistency score includes: Candidate Tibetan sentences and candidate Chinese sentences are encoded separately by a pre-set multilingual sentence vector encoder to obtain the second Tibetan sentence vector and the second Chinese sentence vector; Obtain the second cosine similarity between the second Tibetan sentence vector and the second Chinese sentence vector; Based on the second cosine similarity, a cross-language sentence vector matching score is obtained corresponding to the candidate Tibetan-Chinese sentence pair; The candidate Tibetan sentences and candidate Chinese sentences are classified into different domains by a pre-trained domain classifier to obtain the domain classification results. The domain classification results are then scored to obtain the domain relevance score corresponding to the candidate Tibetan-Chinese sentence pair. The first perplexity of the candidate Tibetan sentence is calculated using a preset Tibetan language perplexity model, and the second perplexity of the candidate Chinese sentence is calculated using a preset Chinese language perplexity model. Based on the first perplexity and the corresponding second perplexity, the language fluency score corresponding to the candidate Tibetan-Chinese sentence pair is obtained. The comprehensive quality score for each candidate Tibetan-Chinese sentence pair is obtained by weighted summation of the vocabulary coverage score, bidirectional translation consistency score, cross-language sentence vector matching score, domain relevance score, and language fluency score. Tibetan-Chinese sentence pairs with a comprehensive quality score higher than a preset threshold are used as the Tibetan-Chinese parallel corpus.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the Tibetan language large language model training method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the Tibetan language large language model training method as described in any one of claims 1 to 8.

Images