Data selection method and device for large language model oriented to security optimization and efficient training

Abstract

The application relates to the field of large language models, in particular to a large language model data selection method and device for security optimization and efficient training. The method comprises the following steps: modeling a data selection process as a double-layer optimization problem, wherein the double-layer optimization comprises inner-layer optimization and outer-layer optimization; constructing a data selector with security perception capability, sorting original data based on the data selector, and preferentially selecting a core data subset with high quality and safety key; using the core data subset as input data in combination with a preset alignment technology for security alignment training of the large language model; and performing final training on the basis of the aligned model using fine-tuning data provided by a user to obtain a target model with security guarantee. The application significantly reduces the training time while improving the model inference performance and security defense capability, and solves the problems of high dependence on data quality and large calculation cost of existing defense methods.

Description

Technical Field

[0001] This invention belongs to the technical field of large language models, specifically relating to a method and apparatus for selecting large language model data for security optimization and efficient training. Background Technology

[0002] Large language models are a class of deep generative models trained on massive corpora, demonstrating powerful generative capabilities in tasks such as question-answering, text writing, and code generation. In recent years, with the continuous expansion of model size and training data volume, large language models have been widely deployed in numerous application scenarios, such as automated writing systems, intelligent question-answering platforms, and intelligent assistants. However, unaligned large language models may generate inappropriate content after receiving harmful or adversarial input instructions, posing serious security risks. Therefore, secure alignment of the model before deployment has become a crucial step in ensuring its security.

[0003] To improve the output security of large language models, existing alignment methods mainly fall into three categories: Supervised Fine-tuning (SFT), Reinforcement Learning Based on Human Feedback (RLHF), and Direct Preference Optimization (DPO). These methods guide the model to learn behavioral patterns that conform to human preferences by introducing high-quality, safe training samples, thereby effectively suppressing harmful or inappropriate outputs. However, recent research has found that even aligned large language models may forget their original safety constraints when faced with fine-tuned data containing harmful information provided by users, leading to the "harmful fine-tuning" problem, which seriously affects the model's security performance after deployment. To address this issue, existing research has proposed a series of defensive methods at the alignment stage, mainly including T-Vaccine, RepNoise, and Booster. These methods enhance the model's robustness to potentially harmful instructions by injecting adversarial training samples or introducing specific regularization mechanisms during the alignment stage, thereby improving its overall defense capability.

[0004] However, existing defense methods generally suffer from two key problems: (1) they are highly dependent on the quality of the original alignment data. If there are many redundant or low-quality samples in the data, the model is prone to a decline in security defense performance; (2) as the scale of alignment data increases, training time and resource consumption increase significantly, leading to a substantial increase in training costs. For example, when the number of alignment samples in the T-Vaccine method increases from 1,000 to 5,000, its training time increases from 0.3 hours to 3.08 hours, an increase of nearly tenfold.

[0005] In summary, existing large language model security alignment defense methods have made some progress in improving model robustness and security performance, but they still struggle to simultaneously achieve both defense effectiveness and training efficiency. On the one hand, these methods heavily rely on the quality of the alignment data; the presence of noise or non-critical samples in the original alignment data reduces the model's security defense capability. On the other hand, as the scale of the alignment data increases, the computational resources and time costs required for training rise significantly, severely limiting the scalability and deployment efficiency of defense methods in practical applications. Therefore, how to select a core subset from the original alignment data that combines high quality and security criticality has become a crucial problem that urgently needs to be solved. Summary of the Invention

[0006] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method and apparatus for selecting large language model data for security optimization and efficient training. By training a data selector with security awareness, the samples in the original alignment dataset are sorted, and a high-quality and security-critical subset of data is automatically selected to construct a compact and efficient core alignment dataset.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for selecting large language model data for security optimization and efficient training, comprising the following steps:

[0009] The data selection process is modeled as a two-level optimization problem, which includes inner-level optimization and outer-level optimization.

[0010] A data selector with security awareness is constructed. Based on the data selector, the original data is sorted, and a high-quality and security-critical core data subset is selected first. The construction of the data selector with security awareness includes: applying the chain rule to the outer layer optimization objective function to construct the update formula of the data selector parameter w; using a first approximate gradient descent to obtain the model parameters for the inner layer optimization problem; and substituting the inner layer approximate solution into the data selector update formula to obtain the final w update rule.

[0011] The core data subset is used as input data and combined with a preset alignment technique for secure alignment training of a large language model.

[0012] Based on the aligned model, the final training is performed using user-provided fine-tuning data to obtain a target model with security guarantees.

[0013] As a preferred technical solution, the data selection process is modeled as a two-level optimization problem, specifically as follows:

[0014] Let the training dataset be... Where x i y represents the i-th input instruction. i Indicate the corresponding standard output label; the dataset Divided into training set With the validation set And introduce harmful datasets Used to simulate potential security risks faced by the model;

[0015] To model the data selection process, we construct the following two-level optimization problem:

[0016]

[0017] Where f(.) represents the value in the validation set. Loss of experience Indicates a harmful dataset The gradient of the harmful loss calculated on the training set, g(θ) represents the gradient of the harmful loss calculated on the training set. The experience loss on γ(w) = (γ1(w), ..., γ i (w),…,γ N (w) represents the data selection function, implemented using the softmax form: α is the step size.

[0018] As a preferred technical solution, the method of applying the chain rule to the outer layer optimization objective function to construct the update formula for the selector parameter w is as follows:

[0019] A gradient-based optimization method is used to iteratively update w, and the update rule is as follows:

[0020]

[0021] Where, η w The learning rate for the data selector. This represents the gradient on the validation set, where α is the step size. Represents the gradient on harmful datasets. This represents the gradient on the training set.

[0022] As a preferred technical solution, it also includes approximating and simplifying higher-order gradient terms to reduce computational overhead, specifically:

[0023] The second derivative in the formula will be updated. Approximating this term as a constant, the simplified update formula is:

[0024]

[0025] As a preferred technical solution, the model parameters θ are obtained by first-order approximate gradient descent for the inner-layer optimization problem. *(w), specifically:

[0026]

[0027] Where θ0 is the initial model obtained by training several steps on the original aligned data, and η θ For model learning rate, Let θ0 be the gradient of θ0 on the training set.

[0028] As a preferred technical solution, the inner approximate solution is substituted into the selector update formula to obtain the final w update rule, specifically:

[0029] according to The update formula for the selector is as follows:

[0030]

[0031] As a preferred technical solution, the process of sorting the original data based on a data selector and prioritizing the selection of a high-quality and security-critical core data subset specifically involves:

[0032] Iterative training is performed on the data selector parameters w;

[0033] The original dataset is sorted based on the trained data selector, where the weight value γ i A larger (w) indicates that the sample is of higher quality and more critical to safety; the weight value γ i The smaller the (w) value, the weaker the robustness of the sample to the model during alignment training;

[0034] The core subset is formed by selecting the top K samples by weight from the ranking results. This is used for alignment fine-tuning in subsequent models.

[0035] Secondly, the present invention provides a large language model data selection system for security optimization and efficient training, which is applied to the large language model data selection method for security optimization and efficient training, including a data modeling module, a data selector construction module, a first training module and a second training module.

[0036] The data modeling module is used to model the data selection process as a two-layer optimization problem, which includes inner-layer optimization and outer-layer optimization.

[0037] The data selector construction module is used to construct a data selector with security awareness capabilities. Based on the data selector, the original data is sorted, and a high-quality and security-critical core data subset is selected first. The construction of the data selector with security awareness capabilities includes: applying the chain rule to the outer layer optimization objective function to construct the update formula for the data selector parameter w; using a first approximate gradient descent to obtain the model parameters for the inner layer optimization problem; and substituting the inner layer approximate solution into the data selector update formula to obtain the final w update rule.

[0038] The first training module is used to take the core data subset as input data and combine it with a preset alignment technique for secure alignment training of a large language model.

[0039] The second training module is used to perform final training based on the aligned model using user-provided fine-tuning data to obtain a target model with security guarantees.

[0040] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0041] At least one processor; and,

[0042] A memory communicatively connected to the at least one processor; wherein,

[0043] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to execute the large language model data selection method for security optimization and efficient training.

[0044] Fourthly, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the aforementioned method for selecting large language model data for security optimization and efficient training.

[0045] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0046] (1) This invention proposes a data selection method for large language models that is geared towards security optimization and efficient training. It can effectively alleviate the problems of existing defense methods being highly dependent on the quality of aligned data and having high training costs, and improve the robustness and defense capability of the model when facing harmful fine-tuning.

[0047] (2) This invention constructs a security-aware data selector, models the data selection process as a two-layer optimization problem, learns to identify high-quality and security-critical samples based on simulated adversarial examples and high-quality responses, and ensures that the selected data subset has high alignment effectiveness and security.

[0048] (3) This invention proposes a data sorting and filtering mechanism that can automatically select a high-quality core subset from the original large-scale aligned data, significantly reducing the scale of training data and effectively reducing training time while maintaining or improving the model's defense and inference performance.

[0049] (4) This invention can be seamlessly integrated with existing mainstream alignment methods (such as SFT, T-Vaccine, RepNoise, etc.), and has good versatility and compatibility. It exhibits excellent defense effect and inference performance improvement in multiple model architectures and attack scenarios, and is suitable for the security fine-tuning deployment of large language models in real-world scenarios. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating a method for selecting large language model data for security optimization and efficient training, according to an embodiment of the present invention.

[0052] Figure 2 This is a block diagram of a large language model data selection system for security optimization and efficient training according to an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the electronic device according to an embodiment of the present invention. Detailed Implementation

[0054] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0055] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0056] Please see Figure 1 This embodiment proposes a data selection method for large language models aimed at security optimization and efficient training, which includes the following steps:

[0057] S1. Model the data selection process as a two-level optimization problem.

[0058] Furthermore, step S1 specifically includes the following:

[0059] S101. Let the training dataset be... Where x i y represents the i-th input instruction. i This represents the corresponding standard output label. (Dataset) Divided into training set With the validation set And assume that a harmful dataset is also present. This is used to simulate potential security risks faced by the model.

[0060] S102. Based on the above settings, construct the following two-level optimization problem to model the data selection process:

[0061]

[0062] Where f(.) represents the value in the validation set. Loss of experience Indicates a harmful dataset The gradient of the harmful loss calculated on the training set, g(θ) represents the gradient of the harmful loss calculated on the training set. The experience loss on γ(w) = (γ1(w), ..., γ i (w),…,γ N (w) represents the data selection function, implemented using the softmax form: α is the step size. Intuitively, the inner optimization function aims to optimize the model parameters by training on a weighted subset of the training data (i.e., the selected core subset), thereby obtaining the model θ. * (w). In the outer optimization, the weights w of the selector are optimized to identify a subset of training cores that can still generalize well after harmful gradient perturbations, and to achieve robust generalization performance on the validation set.

[0063] S2. Construct a data selector with security awareness capabilities, sort the original data based on the data selector, and prioritize the selection of high-quality and security-critical core data subsets.

[0064] Furthermore, step S2 specifically includes the following:

[0065] S201. Apply the chain rule to the outer layer optimization objective function to construct the update formula for the selector parameter w.

[0066] Gradient-based optimization methods (such as SGD) are used to iteratively update w, and the update rules are as follows:

[0067]

[0068] Where, η w The learning rate for the data selector. This represents the gradient on the validation set, where α is the step size. Represents the gradient on harmful datasets. This represents the gradient on the training set.

[0069] S202, Approximate simplification of higher-order gradient terms to reduce computational overhead.

[0070] Due to the second derivative in the updated formula This requires significant computational resources, so we approximate this term as a constant, thus simplifying the update formula to:

[0071]

[0072] S203. The model parameters θ are obtained by using a first approximate gradient descent to solve the inner layer optimization problem. * (w), as follows:

[0073]

[0074] Where θ0 is the initial model obtained by training several steps on the original aligned data, and η θ This is the model learning rate. Since some LoRA parameters are initially set to zero, this initialization helps stabilize the training process. Let θ0 be the gradient of θ0 on the training set.

[0075] S204. Substitute the inner approximate solution into the selector update formula to obtain the final w update rule.

[0076] according to The update formula for the selector is as follows:

[0077]

[0078] As shown in Algorithm 1, this embodiment provides the specific algorithm steps for selecting the selector.

[0079] Algorithm 1: Security-Aware Data Selection Algorithm

[0080]

[0081] S205. Based on Algorithm 1, the selector parameter w is trained, and then the original dataset is sorted based on the selector to obtain a high-quality and safety-critical subset of data with weight γ. i The larger the value of (w), the higher the quality and safety criticality of the sample; the weight value γ i The smaller (w) is, the weaker the sample's robustness to the model during alignment training, and it may even be a redundant sample.

[0082] S206. Accordingly, select the K samples with the highest weight values ​​from the ranking results to form the core subset. This is used for efficient and safe alignment fine-tuning of subsequent models. This core subset has both training and safety value, significantly reducing training costs while effectively enhancing the model's robustness and output safety in the face of malicious instructions.

[0083] S3. Using this core subset as input data, combine it with mainstream alignment techniques for secure alignment training of large language models.

[0084] Furthermore, the mainstream alignment techniques include at least one of Supervised Fine-Tuning (SFT), Human Feedback-Based Reinforcement Learning (RLHF), Direct Preference Optimization (DPO), T-Vaccine, RepNoise, or Booster.

[0085] S4. Based on the aligned model, perform final training using user-provided fine-tuning data to obtain a target model with security guarantees.

[0086] In another embodiment of the present invention, the feasibility of the present invention was verified, mainly including the following five steps:

[0087] S1. Experiment setup, details are as follows:

[0088] Datasets and Models: This embodiment uses two widely used alignment datasets: the BeaverTails dataset and the RepNoise-Refusal dataset. Simultaneously, v benign data points are selected from BeaverTails to form a validation dataset, and v harmful data points are selected to form a harmful dataset. For fine-tuning tasks, to simulate diverse user scenarios, this invention selects three representative fine-tuning tasks: SST2 text sentiment classification, GSM8K mathematical reasoning, and AGNEWS news text classification. To simulate harmful fine-tuning attacks in a real-world environment, during the fine-tuning phase, this invention injects harmful data sampled from the BeaverTails dataset in proportion p into the user fine-tuning data, mixing it with the remaining proportion (1-p) of normal task data to form a fine-tuning sample set of n data points. This setting is used to evaluate the security robustness of the proposed method during the user fine-tuning phase. For model validation, this invention selected three mainstream pre-trained large language model architectures as evaluation objects: Gemma-2-9B, LLaMA2-7B, and Qwen2.5-7B, to verify the universality and effectiveness of the proposed method across different model architectures. Unless otherwise specified, the default experimental parameters in this invention are set as follows: number of aligned samples m = 5000, and validation and harmful dataset sampling v = 200. All experiments were conducted on a GPU server equipped with an A100 processor with 80GB of VRAM.

[0089] Benchmark Methods: This embodiment compares the proposed data selection method, Pharmacist, with three existing benchmark methods: ALL (using all samples in the original alignment dataset), Random (randomly selecting m samples from the original alignment dataset), and TaskVary (currently the only method for alignment data selection, aiming to select samples with low similarity to the user's fine-tuning task). Furthermore, to verify the versatility and enhancement effect of Pharmacist, this embodiment also conducts combined experiments with five mainstream alignment stage methods (SFT, Vaccine, RepNoise, Booster, and T-Vaccine) to comprehensively evaluate its overall performance in improving model output security and optimizing training efficiency.

[0090] S2. Performance comparison on aligned datasets of different complexities, details are as follows:

[0091] Please refer to Table 1. This example demonstrates the generalization ability of the Pharmacist method on alignment datasets of varying complexity. Specifically, on the RepNoise-Refusal dataset, the ALL method uses all 5018 samples for the alignment training stage, while Random, TaskVary, and Pharmacist each select 2500 samples according to their respective strategies and use supervised fine-tuning (SFT) to align the models. Subsequently, all aligned models are fine-tuned on the GSM8K dataset. On the BeaverTails dataset, the ALL method uses all 11604 samples, while the other three methods uniformly select 5000 samples for alignment training. Experimental results demonstrate that Pharmacist exhibits superior performance on both datasets: on the RepNoise-Refusal dataset, compared to the ALL, Random, and TaskVary methods, Pharmacist reduces the Harmful Score (HS) by 0.04%, 3.64%, and 3.40%, respectively; on the BeaverTails dataset, Pharmacist achieves the best defense performance across all harmful ratio settings and obtains the highest average fine-tuning accuracy among all benchmark methods. Notably, on the RepNoise-Refusal dataset, Pharmacist outperforms the ALL method using all samples when using only half the data, highlighting its ability to identify high-quality and safety-critical samples. Although Pharmacist's defense capability is slightly lower than the model using all data at certain harmful ratio settings, its average training efficiency improvement exceeds 50%, fully demonstrating Pharmacist's excellent balance between defense performance and computational efficiency.

[0092] Table 1 Performance comparison on aligned datasets of different complexities

[0093]

[0094] S3. Performance analysis on different fine-tuning datasets, details of which are as follows:

[0095] This embodiment demonstrates the generalization ability of the Pharmacist method on different fine-tuning datasets in Table 2. Taking Llama2-7B as an example, three representative fine-tuning datasets (SST2, AGNEWS, and GSM8K) were selected to evaluate T-Vaccine, and different data selection strategies were compared. Experimental results show that Pharmacist improves defense performance by an average of 4.50% and 14.80% compared to the two existing data selection baseline methods, Random and TaskVary. Specifically, Pharmacist reduces harmful scores by 3.80%, 4.00%, and 5.70% on GSM8K, AGNEWS, and SST2, respectively, and by 16.90%, 11.80%, and 15.70% compared to TaskVary. Furthermore, Pharmacist achieves the highest average fine-tuning accuracy among all baseline methods, further validating the effectiveness of its selected data subset. Notably, Pharmacist outperforms the ALL method, which uses fully aligned data, on multiple tasks. For example, in the GSM8K task, Pharmacist reduced harmful scores by 1.2% and improved fine-tuning accuracy by 0.9% compared to ALL. These results demonstrate that the Pharmacist method not only significantly improves the robustness and safety of the model, but also has good task generalization ability, especially showing strong performance advantages when facing complex tasks such as GSM8K and AGNEWS.

[0096] Table 2 Performance comparison on different fine-tuning datasets

[0097]

[0098] S4. Performance analysis on different models, details of which are as follows:

[0099] Table 3 of this embodiment demonstrates the generalization ability of the Pharmacist method on different pre-trained models. In addition to the aforementioned experiments based on LLaMA2-7B, this embodiment further evaluates the defensive performance of Pharmacist on two state-of-the-art mainstream large language model architectures (Gemma-2-9B and Qwen2.5-7B). In each set of experiments, the same model was consistently used in all three stages: data selection, alignment training, and user fine-tuning (e.g., when evaluating Gemma-2-9B, it was used in all three stages). Experimental results show that Pharmacist improves the average defensive performance by 3.50% and 16.86% compared to existing data selection methods Random and TaskVary, respectively. Specifically, compared to Random, Pharmacist reduced harmful scores by 2.70%, 3.80%, and 4.00% on Gemma-2-9B, LLaMA2-7B, and Qwen2.5-7B models, respectively; and by 16.20%, 16.90%, and 17.50% compared to TaskVary. Furthermore, Pharmacist achieved the highest average fine-tuning accuracy across all models, further validating its effectiveness across different model architectures. Notably, Pharmacist even outperformed the ALL strategy using fully aligned data in most scenarios. For example, on Gemma-2-9B, Pharmacist only achieved a 0.60% higher harmful score than the ALL method, but improved fine-tuning accuracy by 3.3% and reduced training time by 59.65%. These results demonstrate that the Pharmacist method not only possesses significant advantages in defensive effectiveness but also exhibits excellent computational efficiency and maintains good robustness and generalization ability on larger, more complex language models.

[0100] Table 3 Performance comparisons of different models

[0101]

[0102] Performance analysis of S5 and Pharmacist when used in conjunction with existing alignment phase defense technologies, details of which are as follows:

[0103] Table 4 of this embodiment illustrates the performance of the Pharmacist method when combined with existing alignment-stage defense techniques. To further verify the effectiveness of Pharmacist, alignment datasets selected by four different data selection methods (including Random, TaskVary, ALL, and Pharmacist) were applied to five mainstream alignment-stage defense techniques: SFT, RepNoise, Booster, Vaccine, and T-Vaccine. Experimental results show that compared with the commonly used random sampling strategy (Random), Pharmacist significantly improves the performance of various defense methods: improving the defense effectiveness of SFT (+3.80%), RepNoise (+2.00%), Booster (+1.70%), Vaccine (+3.50%), and T-Vaccine (+1.10%), respectively. It also improves the inference performance of each method by 1.00%, 1.30%, 1.80%, 2.64%, and 0.70%, respectively. Compared to TaskVary, Pharmacist achieves stronger defense capabilities by combining various defense techniques. For example, in the RepNoise method, Pharmacist delivers an 11.70% improvement in defense performance and a 1.10% improvement in inference performance. Furthermore, compared to using fully aligned data (ALL), Pharmacist not only achieves superior defense performance in most cases but also significantly accelerates training speed. For instance, when combined with T-Vaccine, Pharmacist delivers a 3.30% improvement in defense performance and a 1.10% improvement in inference performance, while simultaneously increasing training speed by 57.63%.

[0104] Table 4 Performance analysis of Pharmacist when used in conjunction with existing alignment phase defense technologies

[0105]

[0106] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0107] Based on the same idea as the large language model data selection method for security optimization and efficient training in the above embodiments, the present invention also provides a large language model data selection system for security optimization and efficient training. This system can be used to execute the above-described large language model data selection method for security optimization and efficient training. For ease of explanation, the structural diagram of the embodiment of the large language model data selection system for security optimization and efficient training only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0108] Please see Figure 2 In another embodiment of this application, a large language model data selection system 100 for security optimization and efficient training is provided. The system includes a data modeling module, a data selector construction module, a first training module and a second training module.

[0109] The data modeling module is used to model the data selection process as a two-layer optimization problem, which includes inner-layer optimization and outer-layer optimization.

[0110] The data selector construction module is used to construct a data selector with security awareness capabilities. Based on the data selector, the original data is sorted, and a high-quality and security-critical core data subset is selected first. The construction of the data selector with security awareness capabilities includes: applying the chain rule to the outer layer optimization objective function to construct the update formula for the data selector parameter w; using a first approximate gradient descent to obtain the model parameters for the inner layer optimization problem; and substituting the inner layer approximate solution into the data selector update formula to obtain the final w update rule.

[0111] The first training module is used to take the core data subset as input data and combine it with a preset alignment technique for secure alignment training of a large language model.

[0112] The second training module is used to perform final training based on the aligned model using user-provided fine-tuning data to obtain a target model with security guarantees.

[0113] It should be noted that the large language model data selection system for security optimization and efficient training of the present invention corresponds one-to-one with the large language model data selection method for security optimization and efficient training of the present invention. The technical features and beneficial effects described in the above embodiments of the large language model data selection method for security optimization and efficient training are applicable to the embodiments of the large language model data selection method for security optimization and efficient training. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0114] Furthermore, in the implementation of the large language model data selection system for security optimization and efficient training described above, the logical division of each program module is merely illustrative. In practical applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the large language model data selection system for security optimization and efficient training can be divided into different program modules to complete all or part of the functions described above.

[0115] Please see Figure 3 In one embodiment, an electronic device is provided for implementing a method for selecting large language model data for security optimization and efficient training. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a large language model data selection program 203 for security optimization and efficient training.

[0116] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal storage units and external storage devices of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of the large language model data selection program 203 for security optimization and efficient training, but also to temporarily store data that has been output or will be output.

[0117] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.

[0118] Figure 3 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 3 The structure shown does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0119] The large language model data selection program 203, which is optimized for security and efficient training and stored in the first memory 202 of the electronic device 200, is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0120] The data selection process is modeled as a two-level optimization problem, which includes inner-level optimization and outer-level optimization.

[0121] A data selector with security awareness is constructed. Based on the data selector, the original data is sorted, and a high-quality and security-critical core data subset is selected first. The construction of the data selector with security awareness includes: applying the chain rule to the outer layer optimization objective function to construct the update formula of the data selector parameter w; using a first approximate gradient descent to obtain the model parameters for the inner layer optimization problem; and substituting the inner layer approximate solution into the data selector update formula to obtain the final w update rule.

[0122] The core data subset is used as input data and combined with a preset alignment technique for secure alignment training of a large language model.

[0123] Based on the aligned model, the final training is performed using user-provided fine-tuning data to obtain a target model with security guarantees.

[0124] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0125] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0126] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for selecting large language model data for security optimization and efficient training, characterized in that, Includes the following steps: The data selection process is modeled as a two-level optimization problem, which includes inner-level optimization and outer-level optimization. Construct a data selector with security awareness capabilities, sort the raw data based on the data selector, and prioritize the selection of high-quality and security-critical core data subsets; The construction of a data selector with security awareness includes: applying the chain rule to the outer-layer optimization objective function to construct data selector parameters. The update formula is obtained by applying an approximate gradient descent to the inner optimization problem to obtain the model parameters. The approximate solution of the inner layer is then substituted into the data selector update formula to obtain the final result. Update rules; The core data subset is used as input data and combined with a preset alignment technique for secure alignment training of a large language model. Based on the aligned model, final training is performed using user-provided fine-tuning data to obtain a target model with security guarantees; the fine-tuning data includes data used to perform text sentiment classification tasks or news text classification tasks. The data selection process is modeled as a two-level optimization problem, specifically as follows: Let the training dataset be... ,in This represents the i-th input instruction. Indicate the corresponding standard output label; the dataset Divided into training set With the validation set and introduce harmful datasets. Used to simulate potential security risks faced by the model; To model the data selection process, we construct the following two-level optimization problem: in, Indicates in the validation set Loss of experience Indicates a harmful dataset The gradient of the harmful loss calculated above, Indicates in the training set Loss of experience The data selection function is implemented using the softmax form: , Step size; The chain rule is applied to the outer layer optimization objective function to construct the update formula for the selector parameter w, specifically as follows: Gradient-based optimization methods are used to... The iterative update is performed, and the update rules are as follows: in, The learning rate for the data selector. This represents the gradient on the validation set. Step size, Indicates a harmful dataset The gradient of the harmful loss calculated above, This represents the gradient on the training set; It also includes approximating and simplifying higher-order gradient terms to reduce computational overhead, specifically: The second derivative in the formula will be updated. Approximating this term as a constant, the simplified update formula is: ； The model parameters are obtained by using a first approximate gradient descent method for the inner optimization problem. Specifically: in, The initial model parameters are obtained by training the original aligned data for several steps. For model learning rate, for Gradients on the training set; Substituting the inner approximate solution into the selector update formula yields the final result. The update rules are as follows: according to The update formula for the selector is as follows: 。 2. The method for selecting large language model data for security optimization and efficient training according to claim 1, characterized in that, The process of sorting the original data based on a data selector, and prioritizing the selection of a high-quality and security-critical core data subset, specifically involves: Data selector parameters Perform iterative training; The original dataset is sorted based on the trained data selector, where the weight values ​​are... A larger weight indicates a higher quality and greater safety criticality of the sample. The smaller the value, the weaker the robustness of the sample to the model during alignment training; Select the top-ranked results based on their weight values. The core subset of samples This is used for alignment fine-tuning in subsequent models.

3. A large language model data selection system for security optimization and efficient training, characterized in that: The data selection method for large language models for security optimization and efficient training, as applied in any one of claims 1-2, includes a data modeling module, a data selector construction module, a first training module, and a second training module. The data modeling module is used to model the data selection process as a two-layer optimization problem, which includes inner-layer optimization and outer-layer optimization. The data selector construction module is used to construct a data selector with security awareness capabilities, sort the original data based on the data selector, and prioritize the selection of a high-quality and security-critical core data subset. The construction of a data selector with security awareness includes: applying the chain rule to the outer-layer optimization objective function to construct data selector parameters. The update formula is obtained by applying an approximate gradient descent to the inner optimization problem to obtain the model parameters, and then substituting the approximate solution of the inner layer into the data selector update formula to obtain the final result. Update rules; The first training module is used to take the core data subset as input data and combine it with a preset alignment technique for secure alignment training of a large language model. The second training module is used to perform final training based on the aligned model using user-provided fine-tuning data to obtain a target model with security guarantees.

4. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to perform the large language model data selection method for security optimization and efficient training as described in any one of claims 1-2.

5. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the large language model data selection method for security optimization and efficient training as described in any one of claims 1-2.

Images