Large language model training method and device and electronic equipment

Abstract

The invention provides a large language model training method and device and electronic equipment, and relates to the technical field of artificial intelligence such as large models and natural language processing. Comprising the steps that a first sample set of a current classification task is acquired, and the sample set comprises a plurality of text samples and a reference label of each text sample; inputting each text sample into a first large language model, and reasoning a first prediction tag and confidence of each text sample by the first large language model; according to the text length of each text sample, the first prediction label and the confidence coefficient, screening and sorting the first sample set to obtain a plurality of first sample subsets and a training sequence of the first sample subsets; and according to the training sequence, sequentially training a to-be-trained second large language model according to the plurality of first sample subsets. The reliability and compliance of the large language model in a business scene can be improved, it is ensured that the large language model can provide safe and accurate services in a complex application scene, and industry changes are quickly responded.

Description

Technical Field

[0001] This disclosure relates to the field of large model technology, and in particular to the field of artificial intelligence technology such as natural language processing. Background Technology

[0002] With the application of large language models across various industries, their application demands are diversifying across multiple business scenarios. Especially in sensitive areas, ensuring that the model's output complies with industry security requirements has become a key aspect of technological development. When faced with complex and ambiguous topics, the model pre-screens content using security tags to guarantee the legality and security of the generated content. Its core objective is to enable large language models to recognize and adapt to specific industry rules through fine-tuning or reinforcement learning, thereby providing more accurate and compliant security controls in different business scenarios.

[0003] In related technologies, models are typically trained through full parameter fine-tuning. However, due to differences in safety definitions across industries and the continuous changes in industry rules, there are high requirements for the model's dynamic adjustment capabilities and broad adaptability. Summary of the Invention

[0004] This disclosure provides a method, apparatus, and electronic device for training a large language model.

[0005] According to one aspect of this disclosure, a method for training a large language model is provided, comprising: Obtain the first sample set for the current classification task. The sample set includes multiple text samples and a reference label for each text sample. Each text sample in the first sample set is input into the first large language model, and the first large language model infers the first predicted label and confidence level of each text sample. Based on the text length, first predicted label, and confidence level of each text sample, the first sample set is filtered and sorted to obtain multiple first sample subsets and the training order of the first sample subsets. Following the training order, the second large language model to be trained is trained sequentially based on multiple first sample subsets until the training is completed, thus obtaining the target large language model.

[0006] According to another aspect of this disclosure, a training apparatus for a large language model is provided, comprising: The first acquisition module is used to acquire the first sample set for the current classification task. The sample set includes multiple text samples and a reference label for each text sample. The inference module is used to input each text sample in the first sample set into the first large language model, and the first large language model infers the first predicted label and confidence level of each text sample. The second acquisition module is used to filter and sort the first sample set according to the text length, first predicted label and confidence level of each text sample, so as to obtain multiple first sample subsets and the training order of the first sample subsets. The training module is used to train the second large language model to be trained sequentially according to multiple first sample subsets in the training order until the training is completed, so as to obtain the target large language model.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the training method of the large language model described above.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to execute the training method of the large language model described above.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described training method for a large language model.

[0010] This disclosure achieves automatic noise reduction and definition consistency of training data through the semantic confidence mechanism of the first major language model, effectively reducing model performance fluctuations caused by data annotation bias. Compared with methods relying on manual data cleaning or statistical thresholds, this disclosure achieves efficient model-driven data purification, which can significantly reduce the proportion of misleading training samples, effectively avoid catastrophic forgetting, improve training speed, and maintain performance stability on a general security set, achieving complementary alignment of "inheriting general security knowledge and rapidly integrating business definitions".

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram of an exemplary implementation of a training method for a large language model disclosed in this disclosure; Figure 2 This is a schematic diagram of an exemplary implementation of a training method for a large language model disclosed in this disclosure; Figure 3This is a schematic diagram of an exemplary implementation of a training method for a large language model disclosed in this disclosure; Figure 4 This is a schematic diagram of an exemplary implementation of a training method for a large language model disclosed in this disclosure; Figure 5 This is a schematic diagram of a training apparatus for a large language model according to an exemplary embodiment of the present disclosure; Figure 6 This is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data, enabling them to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics through training on large datasets. Their core idea is to learn patterns and structures of natural language through large-scale unsupervised training, thus mimicking the human language cognition and generation process to some extent.

[0015] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Its main applications include machine translation, public opinion monitoring, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, and Chinese OCR.

[0016] Artificial Intelligence (AI) is a key driving force behind the new round of technological revolution and industrial transformation. It is a new technological science that studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. AI is an important component of the discipline of intelligence; it attempts to understand the essence of intelligence and produce a new kind of intelligent machine capable of reacting in a manner similar to human intelligence. AI is a very broad science, encompassing robotics, speech recognition, image recognition, natural language processing, expert systems, machine learning, computer vision, and more.

[0017] Rejection sampling is a random sampling technique based on the Monte Carlo method, used to generate samples from complex target distributions. Its core idea is to use an easily sampled proposal distribution to assist sampling, and to accept or reject candidate samples based on judgment conditions, so that the final accepted samples conform to the target distribution.

[0018] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0019] Figure 1 This is a schematic diagram illustrating an exemplary implementation of a training method for a large language model disclosed in this disclosure, such as... Figure 1 As shown, the method includes the following steps: S101, Obtain the first sample set for the current classification task. The sample set includes multiple text samples and a reference label for each text sample.

[0020] In some implementations, the current classification task is a binary classification task; in some implementations, the current classification task is a multi-classification task; this disclosure does not impose any limitations on this.

[0021] In this embodiment of the disclosure, the first sample set D = {(x(i), y(i))}, where x(i) represents the i-th text sample in the first sample set, and y(i) represents the reference label of the i-th text sample.

[0022] For example, if the current classification task is a binary classification task in a security review scenario, the reference label of the text sample can be a safe label or an unsafe label; optionally, y(i)∈{0,1}, y(i)=0 indicates that the reference label of the i-th text sample is a safe label, and y(i)=1 indicates that the reference label of the i-th text sample is an unsafe label. In other embodiments, the identifier of the safe label or the unsafe label can also be other values, and this disclosure does not limit this.

[0023] For example, if the current classification task is a multi-classification task in a security review scenario, the reference label for the text sample can be a label for a sub-category of the violation type. Similarly, in this embodiment of the disclosure, the labels for the sub-category of the violation type can be assigned values ​​to represent the corresponding reference labels, and this disclosure does not impose any limitations on this.

[0024] S102, input each text sample in the first sample set into the first large language model, and use the first large language model to infer the first predicted label and confidence level of each text sample.

[0025] In some implementations, each text sample is input into a first large language model, which then performs inference on the text sample to obtain a first predicted label and corresponding confidence level. Optionally, the first large language model can be a pre-defined large language model.

[0026] Taking the first major language model as an example of an AI security review model, we can use the AI ​​security review model to reason about the i-th text sample in the first sample set and obtain the first predicted label y'(i) and the corresponding confidence p(i)=p(y'(i)|x(i)).

[0027] S103, based on the text length, first predicted label and confidence level of each text sample, the first sample set is filtered and sorted to obtain multiple first sample subsets and the training order of the first sample subsets.

[0028] In some implementations, the first sample set is filtered based on the first predicted label and confidence level to obtain high-confidence clean samples from the first sample set.

[0029] For example, positive samples in the first sample set can be selected based on whether the first predicted label and the reference label of a text sample are consistent. Then, high-confidence samples among the positive samples can be selected based on their confidence level, ultimately obtaining high-confidence clean samples. In other embodiments, high-confidence samples in the first sample set can be selected based on the confidence level of the text samples. Then, positive samples among the high-confidence samples can be selected based on whether the first predicted label and the reference label are consistent, ultimately obtaining high-confidence clean samples. Here, samples where the first predicted label and the reference label are consistent are considered positive samples.

[0030] In this embodiment of the disclosure, high-confidence clean samples can be sorted according to text length to obtain multiple first sample subsets and the training order of the first sample subsets.

[0031] This disclosure uses confidence level as a data filter to achieve intelligent data purification.

[0032] S104. Following the training order, the second large language model to be trained is trained sequentially based on multiple first sample subsets until the training is completed, thus obtaining the target large language model.

[0033] Following the training sequence, the second language model is trained sequentially using multiple subsets of first samples. For example, the second language model infers the first text sample from the first sample subset, obtains a second predicted label, and compares it with a reference label. If the second predicted label and the reference label are inconsistent, the second language model is adjusted in reverse. Different subsets of first samples correspond to different training difficulty ranges. This mechanism allows the model to learn simple samples first and then complex samples, gradually adapting to changes in semantic boundaries. This process continues until the target large language model is obtained.

[0034] In this embodiment of the disclosure, the first large language model and the second large language model can be the same large language model or different large language models, and this disclosure does not limit this.

[0035] In some implementations, the second major language model can be fine-tuned hierarchically with all parameters. Specifically, a smaller learning rate η1 is used for the bottom encoder parameters, while a larger learning rate η2 is used for the upper classification layer, where η2 >> η1, achieving deep layer preservation and rapid adaptation at the shallow layer. Taking an AI security review model as an example, the model includes a bottom encoder and an upper classifier. The bottom encoder performs deep semantic encoding and contextual understanding on the input text. The upper classifier receives the semantic vector output by the encoder, performs multi-dimensional security risk assessment, and outputs specific risk categories and confidence levels. The bottom layer uses a smaller learning rate for fine-tuning to prevent "forgetting" or "destroying" learned general features, while the upper layer uses a larger learning rate for rapid adjustment to adapt to new tasks.

[0036] In some implementations, a lightweight fine-tuning module can be used, which achieves efficient fine-tuning by injecting a trainable low-rank adaptation matrix into the model, freezing the main model parameters, and requiring only a small number of parameter updates to adapt the large model to downstream tasks.

[0037] Among them, the LoRA module is a parameter optimization technique for efficient fine-tuning of large models. The low-rank adaptation matrix is ​​the core component of the LoRA module, which is a structured parameter matrix that simulates the update of model weights through low-rank decomposition.

[0038] In this embodiment, the first sample set is filtered and sorted according to the text length, first predicted label, and confidence level of each text sample, resulting in multiple first sample subsets and their training order. This disclosure achieves automatic noise reduction and definition consistency of training data through the semantic confidence mechanism of the first large language model, effectively reducing model performance fluctuations caused by data labeling bias and achieving data optimization. This disclosure achieves efficient model-driven data purification, significantly reducing the proportion of misleading training samples. Following the training order, the second large language model to be trained is sequentially trained based on multiple first sample subsets, effectively avoiding catastrophic forgetting, improving training speed, and maintaining performance stability on the general security set, achieving complementary alignment of "inheriting general security knowledge and rapidly integrating business definitions."

[0039] Figure 2 This is a schematic diagram illustrating an exemplary implementation of a training method for a large language model disclosed in this disclosure, such as... Figure 2 As shown, the method includes the following steps: S201, Obtain the first sample set for the current classification task. The sample set includes multiple text samples and a reference label for each text sample.

[0040] S202, input each text sample in the first sample set into the first large language model, and use the first large language model to infer the first predicted label and confidence level of each text sample.

[0041] For a description of steps S201 to S202, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0042] S203, based on the first predicted label and confidence level of each text sample, filter the first sample set to determine multiple first text samples from the first sample set.

[0043] For each text sample in the first sample set, if the reference label of the text sample is the same as the first predicted label, and the confidence of the text sample is greater than a preset confidence threshold, the text sample is determined as the first text sample, which is the high-confidence clean sample. The first text sample is used for stable training.

[0044] In some implementations, a second text sample is obtained from the text samples. The second text sample is rejected to determine a third text sample within the second text sample, and the first text sample is incrementally updated based on the third text sample. The second text sample is used for subsequent review and definition alignment. The second text sample is either a text sample in the first sample set other than the first text sample, or a text sample in the first sample set whose first predicted label and reference label are inconsistent, and whose confidence level is within a preset confidence range.

[0045] In some implementations, the second text sample can be modified or manually screened to obtain a third text sample, and then the first text sample can be incrementally updated based on the third text sample.

[0046] S204, based on the text length of the first text sample, divide the first text sample and its reference label into multiple first sample subsets, and determine the training order of the first sample subsets.

[0047] In some implementations, multiple text length intervals are divided based on the minimum and maximum text lengths of the first text sample. The first text sample and its reference label are divided into multiple first sample subsets based on the text length interval to which the first text sample belongs, wherein different first sample subsets correspond to different text length intervals.

[0048] Wherein, the maximum value of the text length interval corresponding to the first sample subset with training order i is less than the minimum value of the text length interval corresponding to the first sample subset with training order i+1, and i is a positive integer.

[0049] In this embodiment of the disclosure, the training difficulty interval is divided according to the text length L(x) of the first text sample, and different maximum input lengths L are set. max This allows for subsequent training in a progressive manner, following the training order, by sequentially training the second language model based on multiple first sample subsets. For example, there are n first sample subsets, denoted as D1, D2, ..., D... n Then we have L max (D1) <L max (D2)<... <L max (D n ), where L max (D1) represents the maximum text length of each text sample in the first sample subset with training order 1. Similarly, L max (D2) represents the maximum text length of each text sample in the first sample subset with training order 2, and so on, L max (D n () represents the maximum text length of each text sample in the first sample subset with training order n. n is a positive integer.

[0050] In some implementations, the order of the first text samples is obtained based on their text length, and the first text samples and their reference tags are divided into multiple first sample subsets according to the specified number of samples and their order. The specified number of samples corresponding to different first sample subsets may be the same or different, and this disclosure does not impose any limitation on this.

[0051] The training order of the first sample subset is determined based on the order of the first text samples. Specifically, the text length of the first text sample in order j is less than or equal to the text length of the first text sample in order j+1. For example, the first sample subset has n samples, with specified sample numbers Z1, Z2, ..., Z... n Then, according to the order of the first text samples, the first text samples from the 1st to the Z1st and their reference labels are divided into the first sample subset D1 with training order 1, the first text samples from the Z1+1th to the Z1+Z2th and their reference labels are divided into the first sample subset D2 with training order 2, and so on, until the first sample subset D with training order n is obtained. n Where j is a positive integer.

[0052] This disclosure achieves automatic data purification through a confidence level grading mechanism, reducing security fluctuations caused by mislabeled samples, and making the model output more consistent and reliable.

[0053] S205. Following the training order, the second large language model to be trained is trained sequentially based on multiple first sample subsets until the training is completed, thus obtaining the target large language model.

[0054] For a description of step S205, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0055] This embodiment of the disclosure filters the first sample set based on the first predicted label and confidence level of each text sample, determining multiple first text samples from the first sample set. Based on the text length of each first text sample, the first text samples and their reference labels are divided into multiple first sample subsets, and the training order of the first sample subsets is determined. Using a large language model as a semantic evaluator, confidence leveling and data reduction are performed on the vertical training data to achieve error removal, fuzzy correction, and sample refinement, reducing problems such as catastrophic forgetting, data definition conflicts, and fine-tuning instability. By introducing a high-performance security model as a data evaluator and knowledge inheritor, automatic cleaning and grading of training data and multi-stage fine-tuning of the model are achieved, maintaining general security capabilities while enabling rapid adaptation of vertical definitions.

[0056] Figure 3This is a schematic diagram illustrating an exemplary implementation of a training method for a large language model disclosed in this disclosure, such as... Figure 3 As shown, the method includes the following steps: S301, Obtain the first sample set for the current classification task. The sample set includes multiple text samples and a reference label for each text sample.

[0057] S302, input each text sample in the first sample set into the first large language model, and use the first large language model to infer the first predicted label and confidence level of each text sample.

[0058] S303, based on the text length, first predicted label and confidence level of each text sample, the first sample set is filtered and sorted to obtain multiple first sample subsets and the training order of the first sample subsets.

[0059] For a description of steps S301 to S303, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0060] S304 adds a trainable low-rank adaptation matrix to the second largest language model.

[0061] In this embodiment, efficient fine-tuning is achieved by injecting a trainable low-rank adaptation matrix into the model. Freezing the main model parameters reduces computational overhead, requiring only a small number of parameter updates to adapt the large model to downstream tasks. In this embodiment, the low-rank adaptation matrix includes a dimensionality reduction matrix A and an increased dimensionality matrix B, which can be embedded in the attention layer of a second large language model or other neural network layers. The dimensionality reduction matrix A projects the input dimension to a low-rank space, while the increased dimensionality matrix B restores the low-rank space features to the original output dimension. By constraining the low-rank nature of the parameter update direction, general features are preserved while efficiently adapting to specific business tasks, achieving lightweight fine-tuning of the large model.

[0062] S305, input the first text samples from multiple first sample subsets into the second large language model in sequence, the second large language model infers the second predicted label of each first text sample, and trains the low-rank fitting matrix based on the second predicted label of the first text sample and the reference label of the first text sample.

[0063] For example, the second language model infers the first text sample in the first sample subset to obtain the second predicted label, and compares the second predicted label with the reference label. If the second predicted label and the reference label are inconsistent, the low-rank adaptation matrix is ​​adjusted.

[0064] S306. Update the initial model parameters of the second largest language model based on the trained low-rank adaptation matrix to obtain the candidate model parameters of the second largest language model under the current classification task.

[0065] Optionally, the updated model parameters can be obtained using the following formula: W’ = W + ɑAB T where W’ represents the candidate model parameters; A and B represent low-rank adaptation matrices, and A, B ∈ d×r , r << d; ɑ represents a scaling factor; and W represents the initial model parameters.

[0066] S307. Continue to obtain the second sample set for the next classification task, and obtain multiple second sample subsets and the training order of the second sample subsets based on the second sample set to train the to-be-trained second large language model, and obtain the candidate model parameters of the second large language model under the next classification task until the candidate model parameters of the second large language model under all classification tasks are obtained.

[0067] In the embodiments of the present disclosure, there are multiple classification tasks. Taking the classification tasks including binary classification tasks and multi-classification tasks, and the current classification task being a binary classification task as an example, after obtaining the candidate model parameters of the second large language model under the binary classification task, in accordance with the same concept as steps S301 to S306, each text sample in the second sample set is input into the first large language model, and the first large language model infers the third prediction label and confidence of each text sample. Based on the text length, third prediction label, and confidence of each text sample, the second sample set is screened and sorted to obtain multiple second sample subsets and the training order of the second sample subsets. The text samples in the multiple second sample subsets are sequentially input into the second large language model, and the second large language model infers the fourth prediction label of each text sample, and trains the low-rank adaptation matrix based on the fourth prediction label and the reference label, and updates the initial model parameters of the second large language model according to the trained low-rank adaptation matrix to obtain the candidate model parameters of the second large language model under the multi-classification task.

[0068] The present disclosure uses the LoRA module for lightweight complementary fine-tuning. Lightweight means that only a small number of parameters need to be updated to complete scene alignment. Complementary fine-tuning means that model fine-tuning can achieve the complementarity of general security capabilities and business definitions, avoid the problem of catastrophic forgetting, and maintain the general protection capabilities of the model in the long term. It supports minute-level updates and modular deployment, can quickly respond when industry rules or business strategies change, and does not require re-training in full. The number of fine-tuning parameters is small, the computational resource requirements are low, it can be quickly iterated in the existing hardware environment, and the model maintenance cost is greatly reduced. The present disclosure enables the product to improve the security accuracy rate, update efficiency, reduce the operation cost, and provide a more robust and controllable basic ability for security alignment.

[0069] S308: Perform parameter fusion on the candidate model parameters of the second largest language model under each classification task to obtain the target model parameters of the target large language model.

[0070] In this embodiment of the disclosure, the candidate model parameters of the second largest language model under each classification task can be weighted to obtain the target model parameters of the target large language model.

[0071] Optionally, taking classification tasks, including binary classification and multi-class classification, as an example, the target model parameters can be obtained using the following formula: W' fusion =λ1W' binary +λ2W' multi Among them, W' multi W' represents the parameters of the candidate language model for the second largest classification task. binary W' represents the parameters of the candidate language model for the second largest language classification task. fusion λ1 and λ2 represent the target model parameters.

[0072] In this embodiment, a lightweight LoRA fine-tuning and multi-task fusion mechanism is utilized to achieve efficient adaptation of business security definitions while retaining general security capabilities. This achieves a complementary alignment that "retains foundational security knowledge and quickly injects business rules." This embodiment can implement a multi-stage security alignment scheme that enables data self-optimization, fine-tuning to prevent forgetting, and rapid definition adaptation, thereby meeting the dynamic security needs of large models in multiple industries and scenarios.

[0073] Figure 4 This is a schematic diagram illustrating an exemplary implementation of a training method for a large language model disclosed in this disclosure, such as... Figure 4As shown in this embodiment, a first sample set for the current classification task is obtained. The sample set includes multiple text samples and a reference label for each text sample. Each text sample is input into a first large language model, which infers the first predicted label and confidence level for each text sample. Based on the first predicted label and confidence level of each text sample, the first sample set is filtered to determine multiple first text samples. Based on the text length of the first text samples, the first text samples and their reference labels are divided into multiple first sample subsets, and the training order of the first sample subsets is determined. A trainable low-rank adaptation matrix is ​​added to a second large language model. The first text samples from the multiple first sample subsets are sequentially input into the second large language model, which infers the second predicted label for each first text sample. The low-rank adaptation matrix is ​​trained based on the second predicted label and the reference label of the first text sample. The initial model parameters of the second large language model are updated based on the trained low-rank adaptation matrix to obtain the candidate model parameters of the second large language model for the current classification task. Continue to obtain the second sample set for the next classification task, and obtain multiple second sample subsets and their training order based on the second sample set to train the second language model to be trained, so as to obtain the candidate model parameters of the second language model under the next classification task, until the candidate model parameters of the second language model under all classification tasks are obtained; perform parameter fusion on the candidate model parameters of the second language model under each classification task to obtain the target model parameters of the target language model.

[0074] In some implementations, a general security test set and a business-specific test set can be obtained. The target large language model can be tested based on the general security test set and the business-specific test set to obtain multiple comparative evaluation metrics, such as precision, recall, false positive rate, and definition consistency metrics. The model can be continuously evaluated and dynamically iterated and optimized based on the comparative evaluation metrics.

[0075] The general security test set is used to verify whether the general security capabilities of the model are maintained, while the business-specific test set includes the "false positive set", "false negative set", and "gray zone sample set" under specific business rules.

[0076] In implementation, the original and improved models can be deployed simultaneously in online business systems. Based on real user interaction logs, changes in false positive rates and user satisfaction feedback are statistically analyzed to achieve continuous verification and parameter self-tuning in real-world scenarios. Through closed-loop evaluation and retraining, the large language model achieves dynamic adaptive evolution. While maintaining general capabilities, the model can quickly respond to changes in new regulations or business definitions, enabling minute-level security policy updates and deployments.

[0077] This disclosure addresses the issues of long policy update cycles and poor adaptability in existing security models by forming a dynamic evaluation closed loop through multi-task LoRA merging and test feedback. In terms of computational performance, this method requires very few parameter updates, significantly reduces storage requirements, and can achieve minute-level training and deployment in general environments, significantly improving the update speed and maintainability of security models.

[0078] The following is an introduction to the application scenarios and application areas of this disclosure: For example, in content security and intelligent moderation scenarios, text samples can be comment texts from short videos, social media platforms, online education, and other fields. Tags (such as the first predicted tag obtained by the first large language model inferring the text sample, or the second predicted tag obtained by the second large language model inferring the text sample, or the reference tag of the text sample) are security classification tags. In this scenario, the large language model training method provided in this disclosure can quickly calibrate the model's judgment criteria for boundary content, improve recognition accuracy, and reduce false positives and false negatives.

[0079] In financial risk control and compliance review scenarios, text samples can be texts related to transactions and investment promotions, with tags indicating whether they comply with or do not meet regulatory requirements. In this scenario, the training method for the large language model provided in this disclosure can be used to train a large financial model to understand local regulatory rules, securely aligning it with transaction and investment promotion scenarios to ensure that the output complies with regulatory requirements.

[0080] In medical and privacy protection scenarios, text samples can be question-and-answer texts from fields such as medical Q&A and health consultations, tagged with privacy information judgment tags or sensitive information judgment tags. In this scenario, the training method of the large language model provided in this disclosure can be combined with industry-defined fine-tuning standards for judging privacy and sensitive information in the model to ensure content compliance and privacy security.

[0081] In the enterprise security assistant and customer service model scenario, text samples can be internal knowledge base text or question-and-answer text from a large customer service model, with labels representing semantic boundaries. In this scenario, the large language model training method provided in this disclosure can be combined with the enterprise's specific brand security strategy and semantic boundary definition to achieve security protection capabilities that coexist across multiple scenarios.

[0082] This disclosure, while ensuring the general defense capabilities of the open-source security model, enables rapid alignment of business definitions and inheritance of security knowledge, significantly improving the model's security, controllability, and deployment flexibility.

[0083] This disclosure presents a secure alignment algorithm for large language models based on multi-stage data optimization and lightweight complementary fine-tuning. It boasts high versatility and scalability, and can be widely applied in areas such as content security, financial compliance, medical privacy, and enterprise knowledge protection. The algorithm obtains a high-quality core dataset by performing confidence grading and automatic cleaning on the training data to remove erroneous and ambiguous samples. Then, it utilizes LoRA lightweight fine-tuning to achieve rapid integration of general security capabilities with business definitions, avoiding catastrophic omissions.

[0084] In content security scenarios, it can be used for security definition calibration of social media platforms, short video platforms, and educational review models, significantly reducing false positives and false negatives. In the financial and government sectors, it can quickly inject the latest regulatory rules and definitions, achieving minute-level updates. In medical and enterprise scenarios, it can ensure that privacy and sensitive information are not leaked by the model. The algorithm is highly efficient in training and flexible in deployment, and can be embedded as a general-purpose security alignment engine into existing large model systems, providing stable, low-cost, and controllable security compliance capabilities for different industries.

[0085] Figure 5 This is a schematic diagram of a training apparatus for a large language model according to an exemplary embodiment of the present disclosure, such as... Figure 5 As shown, the training device 500 for the large language model includes: The first acquisition module 510 is used to acquire the first sample set for the current classification task. The sample set includes multiple text samples and reference labels for each text sample. The inference module 520 is used to input each text sample in the first sample set into the first large language model, and the first large language model infers the first predicted label and confidence level of each text sample. The second acquisition module 530 is used to filter and sort the first sample set according to the text length, first predicted label and confidence of each text sample to obtain multiple first sample subsets and the training order of the first sample subsets. Training module 540 is used to train the second large language model to be trained in the order of training, based on multiple first sample subsets, until the training is completed, and obtain the target large language model.

[0086] In some implementations, the second acquisition module 530 is further configured to: Based on the first predicted label and confidence level of each text sample, the first sample set is filtered to determine multiple first text samples from the first sample set; Based on the text length of the first text sample, the first text sample and its reference label are divided into multiple first sample subsets, and the training order of the first sample subsets is determined.

[0087] In some implementations, the second acquisition module 530 is further configured to: For each text sample in the first sample set, if the reference label of the text sample is the same as the first predicted label and the confidence of the text sample is greater than the preset confidence threshold, the text sample is determined to be the first text sample.

[0088] In some implementations, the second acquisition module 530 is further configured to: Reject sampling of the second text sample to identify the third text sample within the second text sample; Update the first text sample based on the third text sample; Wherein, the second text sample is a text sample in the first sample set other than the first text sample, or the second text sample is a text sample in the first sample set whose first predicted label and reference label are inconsistent and whose confidence level is within the preset confidence level range.

[0089] In some implementations, the second acquisition module 530 is further configured to: Based on the minimum and maximum text lengths of the first text sample, divide the text into multiple length ranges; Based on the text length range to which the first text sample belongs, the first text sample and its reference label are divided into multiple first sample subsets, where different first sample subsets correspond to different text length ranges.

[0090] In some implementations, the maximum value of the text length interval corresponding to the first sample subset with training order i is less than the minimum value of the text length interval corresponding to the first sample subset with training order i+1, where i is a positive integer.

[0091] In some embodiments, the second acquisition module 530 is further used for Based on the text length of the first text sample, obtain the order of the first text samples; Based on the specified number of samples, the first text samples and their reference labels are divided into multiple first sample subsets according to the order of the first text samples.

[0092] In some implementations, the training order of the first sample subset is determined according to the order of the first text samples, wherein the text length of the first text sample in order j is less than or equal to the text length of the first text sample in order j+1, and j is a positive integer.

[0093] In some implementations, the training module 540 is also used for: Add a trainable low-rank adaptation matrix to the second largest language model; The first text samples from multiple first sample subsets are sequentially input into the second large language model. The second large language model infers the second predicted label for each first text sample. Based on the second predicted label and the reference label of the first text sample, the low-rank adaptation matrix is ​​trained.

[0094] In some implementations, the training module 540 is also used for: The initial model parameters of the second largest language model are updated based on the low-rank adaptation matrix after training, thus obtaining the candidate model parameters of the second largest language model for the current classification task.

[0095] In some implementations, the classification task has multiple components, and the training module 540 is also used for: Continue to obtain the second sample set for the next classification task, and obtain multiple second sample subsets and their training order based on the second sample set, so as to train the second language model to be trained, and obtain the candidate model parameters of the second language model under the next classification task, until the candidate model parameters of the second language model under all classification tasks are obtained. The parameters of the candidate models of the second largest language model under each classification task are fused to obtain the target model parameters of the target large language model.

[0096] This disclosure achieves automatic noise reduction and definition consistency of training data through the semantic confidence mechanism of the first major language model, effectively reducing model performance fluctuations caused by data annotation bias. Compared with methods relying on manual data cleaning or statistical thresholds, this disclosure achieves efficient model-driven data purification, which can significantly reduce the proportion of misleading training samples, effectively avoid catastrophic forgetting, improve training speed, and maintain performance stability on a general security set, achieving complementary alignment of "inheriting general security knowledge and rapidly integrating business definitions".

[0097] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0098] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0099] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded into random access memory (RAM) 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0100] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0101] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the training method for a large language model. For example, in some embodiments, the training method for a large language model can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the training method for a large language model described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the training method for a large language model by any other suitable means (e.g., by means of firmware).

[0102] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0103] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0104] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0105] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0106] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0107] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0108] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0109] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for a large language model, wherein, include: Obtain the first sample set for the current classification task. The sample set includes multiple text samples and a reference label for each text sample. Each text sample in the first sample set is input into the first large language model, and the first large language model infers the first predicted label and confidence level of each text sample. Based on the text length, the first predicted label, and the confidence level of each text sample, the first sample set is filtered and sorted to obtain multiple first sample subsets and the training order of the first sample subsets. According to the training order, the second large language model to be trained is trained sequentially based on the multiple first sample subsets until the training is completed, and the target large language model is obtained.

2. The method according to claim 1, wherein, The step of filtering and sorting the first sample set according to the text length, the first predicted label, and the confidence level of each text sample to obtain multiple first sample subsets and the training order of the first sample subsets includes: Based on the first predicted label and the confidence level of each text sample, the first sample set is filtered to determine multiple first text samples from the first sample set; Based on the text length of the first text sample, the first text sample and its reference label are divided into multiple first sample subsets, and the training order of the first sample subsets is determined.

3. The method according to claim 2, wherein, The step of filtering the first sample set based on the first predicted label and the confidence level of each text sample to determine multiple first text samples from the first sample set includes: For each text sample in the first sample set, in response to the text sample having the same reference label as the first predicted label and the text sample having a confidence level greater than a preset confidence threshold, the text sample is determined to be the first text sample.

4. The method according to claim 2, wherein, Before dividing the first text sample and its reference label into multiple subsets based on the text length of the first text sample, the method further includes: Reject sampling of the second text sample to determine the third text sample in the second text sample; The first text sample is updated based on the third text sample; Wherein, the second text sample is a text sample in the first sample set other than the first text sample, or the second text sample is a text sample in the first sample set where the first predicted label and the reference label are inconsistent, and the confidence level is within a preset confidence level range.

5. The method according to any one of claims 2-4, wherein, The step of dividing the first text sample and its reference tags into multiple subsets based on the text length of the first text sample includes: Based on the minimum and maximum text lengths of the first text sample, multiple text length intervals are defined. Based on the text length range to which the first text sample belongs, the first text sample and its reference label are divided into multiple first sample subsets, wherein different first sample subsets correspond to different text length ranges.

6. The method according to claim 5, wherein, The maximum value of the text length interval corresponding to the first sample subset with training order i is less than the minimum value of the text length interval corresponding to the first sample subset with training order i+1, where i is a positive integer.

7. The method according to any one of claims 2-4, wherein, The step of dividing the first text sample and its reference tags into multiple subsets based on the text length of the first text sample includes: Based on the text length of the first text sample, obtain the order of the first text samples; Based on the specified number of samples, the first text samples and their reference labels are divided into multiple subsets of the first samples according to the order of the first text samples.

8. The method according to claim 7, wherein, The training order of the first sample subset is determined according to the order of the first text samples, wherein the text length of the first text sample in order j is less than or equal to the text length of the first text sample in order j+1, and j is a positive integer.

9. The method according to any one of claims 1-4, wherein, The step of training the second language model to be trained sequentially based on the plurality of first sample subsets includes: Add a trainable low-rank adaptation matrix to the second largest language model; The first text samples from the plurality of first sample subsets are sequentially input into the second large language model, the second large language model infers the second predicted label for each first text sample, and the low-rank adaptation matrix is ​​trained based on the second predicted label of the first text sample and the reference label of the first text sample.

10. The method according to claim 9, wherein, Also includes: The initial model parameters of the second largest language model are updated based on the low-rank adaptation matrix after training, and the candidate model parameters of the second largest language model under the current classification task are obtained.

11. The method according to claim 10, wherein, The classification task has multiple components. After obtaining the candidate model parameters of the second largest language model under the current classification task, the method further includes: Continue to obtain the second sample set for the next classification task, and obtain multiple second sample subsets and their training order based on the second sample set to train the second language model to be trained, so as to obtain the candidate model parameters of the second language model under the next classification task, until the candidate model parameters of the second language model under all classification tasks are obtained. The candidate model parameters of the second largest language model under each classification task are fused to obtain the target model parameters of the target largest language model.

12. A training device for a large language model, wherein, include: The first acquisition module is used to acquire the first sample set for the current classification task. The sample set includes multiple text samples and a reference label for each text sample. The inference module is used to input each text sample in the first sample set into the first large language model, and to infer the first predicted label and confidence level of each text sample by the first large language model; The second acquisition module is used to filter and sort the first sample set according to the text length of each text sample, the first predicted label and the confidence level, so as to obtain multiple first sample subsets and the training order of the first sample subsets. The training module is used to train the second large language model to be trained sequentially according to the multiple first sample subsets in the training order until the training is completed, so as to obtain the target large language model.

13. An electronic device, wherein, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-11.

Images