Method and device for optimizing large model based on multi-preference alignment and thought chain compression

Abstract

The application provides a large model optimization method and device based on multi-preference alignment and thought chain compression, and relates to the technical field of large models. The method comprises the following steps: optimizing system prompt words of a large model based on thought chain length and accuracy of problems in a test set; constructing a training data set according to the optimized system prompt words, and performing forward fine-tuning on the large model by using the training data set; and performing self-training iteration on the large model according to a plurality of preset preferences to obtain a target large model. The embodiment of the application can reduce the length of the thought chain by optimizing the system prompt words, and can perform self-training iteration on the large model by means of forward fine-tuning and multi-preference alignment, so as to ensure the accuracy and readability of the large model reasoning on the basis of compressing the length of the thought chain.

Description

Technical Field

[0001] This application relates to the field of large model technology, and more specifically, to a method and apparatus for large model optimization based on multi-preference alignment and mind chain compression. Background Technology

[0002] In specific vertical sectors, such as finance, complex reasoning tasks are prevalent in key processes like risk assessment, financial auditing, and compliance checks. These tasks typically require large models to perform multi-step, rigorous, and complex logical reasoning, and to make judgments by integrating knowledge from multiple disciplines such as law, finance, and economics.

[0003] However, while current large-scale reasoning language models possess strong complex reasoning capabilities (e.g., DeepSeekR1, doubao-seed-1.6, OpenAI o1), they often rely on lengthy reasoning chains to enhance accuracy and ensure the integrity and interpretability of the reasoning process when handling such tasks. However, excessively long and redundant reasoning chains not only lead to high computational resource consumption and response latency (multiple studies have shown that large models exceeding a reasonable reasoning length often have relatively lower accuracy), but also introduce unnecessary noise into the reasoning, limiting the deployment and application of these models in real-world financial business scenarios. Summary of the Invention

[0004] This application provides a method and apparatus for optimizing large models based on multi-preference alignment and thought chain compression. This addresses the problems of excessively long thought chains in existing large models, leading to high resource consumption, response latency, and inference noise, thus limiting model application. To achieve this objective, this application provides the following solutions.

[0005] According to one aspect of the embodiments of this application, a large model optimization method based on multi-preference alignment and thought chain compression is provided, the method comprising: The system prompt words of the large model are optimized based on the length of the thought chain and the accuracy of the questions in the test set. A training dataset is constructed based on the optimized system prompts, and the large model is then fine-tuned using the training dataset. The large model is self-trained iteratively based on multiple preset preferences to obtain the target large model.

[0006] In one possible implementation, the system prompt words based on the large model optimized by the length of the thought chain and the accuracy of the questions in the test set include: Determine a large model to be trained, and set the first hyperparameter of the large model to a preset value, wherein the first hyperparameter includes Temperature. The large model is used to predict the problems in the test set multiple times, and the accuracy and corresponding thought chain length are obtained based on the prediction results. The system prompts are adjusted based on the accuracy rate and the length of the thought chain.

[0007] In one possible implementation, constructing the training dataset based on the optimized system prompts includes: The first question set is labeled using multiple labeling models to obtain the labeling results, which include the initial answers corresponding to the first question set. The target answer for the first question set is determined based on the annotation results, and the training dataset is constructed based on the target answer, the first question set, and the optimized system prompts.

[0008] In one possible implementation, determining the target answer for the first question set based on the annotation results includes: Obtain all initial answers for each question in the first question set; The target answer for each question is determined based on the conditions corresponding to the initial answer and the preset voting method, wherein the preset voting method includes majority voting and weighted voting.

[0009] In one possible implementation, the large model is positively fine-tuned using the training dataset, including: Freeze all parameters of the large model and use the training dataset to perform KTO training on the low-rank adapter corresponding to the large model; The trained low-rank adapter is merged with the large model to obtain a new large model.

[0010] In one possible implementation, the self-training iteration of the large model based on multiple preset preferences includes: Identify the current large model to be acquired, and construct a positive and negative preference dataset using the previous generation large model of the current large model and the preferences; Obtain the gold positive sample based on the positive preference dataset of the previous generation large model; The previous generation large model is trained based on the positive and negative preference dataset and the golden positive sample to obtain the contemporary large model.

[0011] In one possible implementation, constructing a positive and negative preference dataset using the previous generation of the current large model and the preferences includes: Set the first parameter of the previous generation large model to a preset value, and use the previous generation large model to predict the problems in the second problem set multiple times. The three-order filtering funnel-style multi-preference annotation uses the prediction results of the previous generation large model to generate the positive and negative preference dataset based on the annotation results. The three-order filtering funnel-style multi-preference annotation includes preference model accuracy annotation, preference thought chain length annotation, and preference reasoning readability annotation.

[0012] In one possible implementation, obtaining the golden positive sample based on the positive preference dataset of the previous generation large model includes: Obtain the data corresponding to each question in the positive preference dataset; The gold standard positive samples are selected based on the accuracy, length, and readability of the data.

[0013] In one possible implementation, training the previous generation large model based on the positive and negative preference dataset and the golden positive sample to obtain the current generation large model includes: Freeze all parameters of the previous generation large model, and train a low-rank adapter of the previous generation large model using the golden positive samples and the positive and negative preference dataset; The trained low-rank adapter and the previous generation large model are merged to obtain the current generation large model.

[0014] According to one aspect of the present application, an electronic device is provided, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above.

[0015] The beneficial effects of the technical solutions provided in this application are: The large model optimization method based on multi-preference alignment and thought chain compression provided in this application includes: optimizing the system prompt words of the large model based on the thought chain length and the accuracy of the questions in the test set; constructing a training dataset based on the optimized system prompt words; using the training dataset to perform positive fine-tuning of the large model; and performing self-training iteration on the large model according to multiple preset preferences to obtain the target large model. The embodiments of this application can reduce the thought chain length by optimizing the system prompt words and self-train iteratively on the large model through positive fine-tuning and multi-preference alignment, thereby ensuring the accuracy and readability of the large model's inference while compressing the thought chain length. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0017] Figure 1 A flowchart illustrating a large model optimization method based on multi-preference alignment and mind chain compression provided in an embodiment of this application; Figure 2 A flowchart illustrating the multi-model collaborative annotation and voting fusion mechanism provided in this application embodiment; Figure 3 A schematic diagram illustrating the construction of the positive and negative preference dataset provided in the embodiments of this application; Figure 4 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0019] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” indicates implementation as “A,” or implementation as “A,” or implementation as “A and B.”

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0021] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0022] The large model optimization method based on multi-preference alignment and mind chain compression provided in this application aims to solve at least one technical problem existing in the prior art.

[0023] In this embodiment, the large model optimization method of this application can be used on servers, computers, and other devices capable of loading large models.

[0024] Optionally, the large model used in this application can be an edge-side large model, which is a large language model that can run directly on terminal devices such as personal computers and automobiles. To accommodate the limited computing power and memory of the devices, its parameter size is usually in the range of 1B to 5B, and it is highly compressed and optimized to achieve low latency, privacy and security, and offline availability while ensuring basic intelligent capabilities.

[0025] Optionally, such as Figures 1-3 As shown, the large model optimization method based on multi-preference alignment and mind chain compression in this application includes: S101: System prompts for large models optimized based on the length of the thought chain and the accuracy of questions in the test set.

[0026] Optionally, a system prompt is a core concept in large language models (LLMs) or dialogue systems. It refers to a pre-defined instruction text provided by the system before the user interacts with the AI. This text is not directly visible to the user but serves as an "internal guide" for the model's operation, defining and controlling the AI's behavior, role, tone, capability boundaries, and response methods. Optimizing system prompts can compress thought chain length, standardize output format, and improve accuracy.

[0027] Optionally, the system prompts for the large model are optimized based on the thought chain length and the accuracy of the questions in the test set. This includes: identifying the large model to be trained; setting the first hyperparameter of the large model to a preset value, where the first hyperparameter includes Temperature; using the large model to predict questions in the test set multiple times; obtaining the accuracy and the corresponding thought chain length based on the prediction results; and adjusting the system prompts based on the accuracy and thought chain length. The number of predictions can be 4, 5, or other numbers.

[0028] Optionally, the large model to be trained can be a NoThink mode (referring to the default mode where the large model directly generates responses, relying on pattern matching and statistical prediction, emphasizing speed and efficiency, and suitable for simple tasks). Compared to a Think mode (referring to the operating mode where the large model enables explicit reasoning mechanisms (such as CoT), emphasizing "slow thinking," and used to improve the accuracy and interpretability of complex tasks), the thinking process is shorter and the thought chain becomes shorter.

[0029] Optionally, Temperature is a core hyperparameter in large language models that controls the randomness and creativity of the output. Specifically, this hyperparameter can be set to 1.0 to maximize the randomness of the model output.

[0030] Optionally, when optimizing system prompts, the readability of the predicted responses can also be detected. Specifically, the "readability of responses" is the average value calculated after multiple predictions for each question in the test set and determining whether the responses are readable. The determination of readability can be aided by constructing prompts using the DeepSeeK-R1 model in a FewShot manner to evaluate the readability of the answers. FewShot (few-shot learning) is a machine learning paradigm whose core goal is to enable the model to quickly adapt to new tasks or identify new categories using a very small number of labeled samples (i.e., "examples"). This application provides examples of negative readability evaluation for directly outputting the answer and negative readability evaluation for semantic incomprehensibility in FewShot.

[0031] Optionally, accuracy is the proportion of samples correctly predicted by the model out of the total number of samples. The thought chain length is the length of the remaining portion of the large model's output after removing relevant answers and formatting.

[0032] In one embodiment, a large model is used to independently predict each question in the test set, and the large model is required to output the prediction results in a specific format (e.g., medical results in the form of $\boxed{answer}$). Specifically, the large model can be controlled to independently make five predictions for each question in the test set. Based on the prediction results, the accuracy (the average of the five predictions' average accuracy), the thought chain length (the average of the five predictions' average thought chain length), and the response readability rate are calculated. The system prompt words are then adjusted based on these accuracy, thought chain length, and response readability rate to obtain a simple and highly accurate optimal system prompt word. The final optimal system prompt word can be: Think step by step, but only keep a minimum draft for each thinking step, with 5 words at most. Note the requirements: Your reply must be as short as possible, no more than 100 characters! - The answer should be output in the form of $\boxed{answer}$. - The answer can only be a single choice, multiple choice, or a calculated numerical result, such as: $\boxed{A}$, $\boxed{BCD}$, etc. Your response must include a reasoning process, which must be as short as possible, not exceeding 100 characters! Furthermore, the reasoning process must be readable, structured, and factual.

[0033] In one embodiment, the large model used for training can be the Qwen3-4B model. By optimizing the prompt words of this model, the Qwen3-4B (Ours) model is obtained. Compared with the original model, this model has a shorter thought process and a standardized output format, which improves the accuracy to a certain extent.

[0034] S102: Construct a training dataset based on the optimized system prompts, and use the training dataset to perform positive fine-tuning on the large model.

[0035] Optionally, a training dataset is constructed based on the optimized system prompts, including: labeling the first question set using multiple labeling models, obtaining the labeling results, the labeling results including the initial answers corresponding to the first question set; determining the target answers for the first question set based on the labeling results, and constructing a training dataset based on the target answers, the first question set, and the optimized system prompts.

[0036] Optionally, when applying the large model to the financial field, the first question set can be a multi-source dataset, covering questions from both business decision-making and academic research domains to meet the three high-frequency needs of enterprises: consulting, retrieval, and task management. The questions in this first question set can be determined according to the domain in which the large model is applied. In order to improve the breadth of the first question set, questions from multiple data sources can be obtained and used to construct the first question set.

[0037] Optionally, multiple annotation models can be used collaboratively to annotate the answers to the question set. Specifically, these multiple annotation models could be Qwen3-235B-A22B, DeepSeek R1, Doubao-1.6 thinking, and Kimi-2, each providing the answers to the questions in the first question set. Using multiple annotation models ensures model heterogeneity, guarantees annotation diversity, and avoids bias from a single model.

[0038] Optionally, the target answer for the first question set is determined based on the annotation results, including: obtaining all initial answers corresponding to each question in the first question set; determining the target answer for each question based on the initial answers and the conditions corresponding to the preset voting method, wherein the preset voting method includes majority voting and weighted voting, and the voting fusion mechanism is realized through the preset voting method.

[0039] In one embodiment, the annotation results include the thought chain and answer output by the annotation model. If multiple annotations give the same answer among multiple answers to the same question, the answer is determined as the target answer based on the majority voting mechanism.

[0040] In one embodiment, if multiple annotation models provide different answers, a weighted voting method is used to determine the target answer. The different annotation models are assigned different weights, and the target answer is determined based on the weights of each annotation model (e.g., the answer output by the annotation with the highest weight is selected as the target answer).

[0041] In one embodiment, when the question in the first problem set is: Which of the following items are tax adjustment reductions? A. Unrecovered losses of general enterprises in previous years (within the previous five years) B. Interest income from government bonds C. Dividend income between qualified resident enterprises D. Tax late payment penalties. Simultaneously, four annotation models—Qwen3-235B-A22B, DeepSeekR1, Doubao-1.6 thinking, and Kimi-2—are used in parallel to answer this question, achieving multi-model collaborative annotation.

[0042] The result of labeling multiple models is: Qwen3-235B-A22B gives: Option A DeepSeek R1 provides: Option A Doubao-1.6 thinking suggests option C. Kimi-2 gives: Option D Based on the majority voting mechanism, option A, which receives more votes, will be marked as the correct answer.

[0043] The result of labeling multiple models is: Qwen3-235B-A22B gives: Option A DeepSeek R1 provides option B. Doubao-1.6 thinking suggests option C. Kimi-2 gives: Option D When different annotation models provide their own answers, a weighted voting method is used for annotation. That is, the model that is more accurate for this type of question is given a higher weight. For example, the weight of the annotation models from high to low is: Doubao-1.6thinking>Qwen3-235B-A22B>Kimi-2>DeepSeek R1.

[0044] In one embodiment, the labeled data (answers) and optimized system prompts are formatted as a training dataset. An example of such a training dataset could be: [{ "messages": [ { "role": "system", "content": "Think step by step, but only keep..."}, { "role": "user", "content": "Please play the role of an expert in the field of finance and accounting..."}, { "role": "assistant", "content": "Interest = Principal × Annual Interest Rate × Deposit Period, $\\boxed{C}$"} ], "label": true },....] Each data point in the training dataset consists of two parts: "messages" and "label." "Messages" represents the dialogue content, and "label" indicates whether the dialogue is acceptable. Within "messages," there are multiple curly braces ({}), each referring to the dialogue content of a specific role. When "role" within the curly brace corresponds to "system," it's a system prompt. When "role" corresponds to "user," it represents the user's question; when "role" corresponds to "assistant," it represents the AI ​​assistant's response. The user's question is the question in the first question set, and the AI ​​assistant's response is the target answer.

[0045] Optionally, the large model can be positively fine-tuned using the training dataset, including: freezing all parameters of the large model, performing KTO training on the low-rank adapter corresponding to the large model using the training dataset, and merging the trained low-rank adapter with the large model to obtain a new large model.

[0046] Optionally, KTO (Kahneman-Tversky Optimization) is a preference optimization method for large language model (LLM) alignment. Its core idea originates from Prospect Theory (proposed by Nobel laureates Daniel Kahneman and Amos Tversky) in behavioral economics. Compared with traditional methods such as DPO and PPO, KTO has significant advantages in terms of data requirements and training efficiency. It is an efficient and economical optimization method for large language models. In the positive supervised fine-tuning stage, this application uses the LLaMA-Factory training framework and employs a training strategy of freezing all parameters of the original large model Qwen3-4B with QLoRA (Quantized LoRA), training only low-rank adapters (LoRAAdapters) to perform KTO positive fine-tuning training on the model. LLaMA-Factory is an open-source, integrated lightweight training framework and fine-tuning platform for LLM (large language models). Its core feature is that it allows users without a strong deep learning background to quickly get started and complete the entire process from data preparation and model fine-tuning to inference deployment. LLaMA-Factory supports a wide range of model architectures, multiple fine-tuning methods, various training tasks, integrates evaluation and inference functions, has comprehensive documentation, and an active community. QLoRA (Quantized Low-Rank Adaptation) is an efficient model fine-tuning technique that combines the LORA (Low-Rank Adaptation) method with deep quantization technology. Typically, the model is first quantized and compressed, and then fine-tuned using LoRA. The core idea of ​​LoRA is to make small-scale, targeted adjustments to the model behavior by adding an additional trainable module (low-rank adapter) without modifying the model parameters.

[0047] In one embodiment, when training with QLoRA, after freezing the parameters of the original large model, KTO training is performed on the low-rank adapters (LoRA Adapters) using the training dataset to obtain qwen3-4B-cot-compression-adapter-v0. Finally, the adapter and the base model (Qwen3-4B, i.e., the frozen parameters of the original model) are modelmerized, merging the incremental weights of the low-rank adapter back into the base model, thus obtaining a new "fully parameterized" model that can be directly inferred without loading additional LoRA modules. This method reduces the number of parameters that need to be trained, and the GPU memory consumed during training is also reduced, effectively reducing GPU memory requirements. The resulting large model can be denoted as qwen3-4B-cot-compression-v0, and its hyperparameters can be: CUDA_VISIBLE_DEVICES=3,1,0,2 nohup llamafactory-cli train \ --stage kto \ --do_train True \ --model_name_or_path . / xinference / modelscope / hub / Qwen / Qwen3-4B \ --preprocessing_num_workers 16 \ --finetuning_type lora \ --template qwen3 \ --flash_attn auto \ --dataset_dir data \ --dataset short_cot_train_only_kto.json \ --cutoff_len 2048 \ --learning_rate 5e-05 \ --num_train_epochs 1000.0 \ --max_samples 100000 --per_device_train_batch_size 2 \ --gradient_accumulation_steps 8 \ --lr_scheduler_type cosine --max_grad_norm 1.0 --logging_steps 5 --save_steps 50 --warmup_steps 0 --packing False --enable_thinking False --report_to none --output_dir saves / Qwen3-4B-Instruct / lora / train_kto_v6_2025-07-26 --bf16 True --plot_loss True --trust_remote_code True --ddp_timeout 180000000 --include_num_input_tokens_seen True --optim adamw_torch --quantization_bit 4 --quantization_method bnb --double_quantization True --lora_rank 16 --lora_alpha 32 --lora_dropout 0.1 --use_rslora True --lora_target all --pref_beta 0.1 --pref_ftx 0 --pref_loss sigmoid& Where: The total number of parameters in the Qwen3-4B (Ours_V0) model is 4,055,498,240, the number of trainable parameters is 33,030,144, and the proportion of trainable parameters is 0.8145%.

[0048] Optionally, during training, the lora parameters are: --lora_rank 16 \ --lora_alpha 32 \ --lora_dropout 0.1 \ --use_rslora True \ --lora_target all \ The `--lora_rank 16` option means that the rank in LoRA is 16, which represents the dimension of the low-rank matrix factorization. A larger value means LoRA can learn more parameters and has a stronger fitting ability, but the computational cost also increases.

[0049] The `--lora_alpha 32` means that the LoRA scaling factor is 32, which controls the intensity of LoRA updates. Here, alpha=32, rank=16 → the scaling factor is 32 / 16=2, meaning the LoRA update will be amplified by a factor of 2.

[0050] The `--lora_dropout 0.1` parameter specifies the percentage of dropout applied in the LoRA layer. To prevent overfitting, dropout is applied to the A matrix output of the LoRA layer. 0.1 indicates that 10% of the neurons are randomly set to zero.

[0051] `--use_rslora True` means: Use RS-LoRA (Rank-Stabilized LoRA) to dynamically adjust the scaling factor α / r of each LoRA layer, making the gradient magnitude of different layers more stable. This improves training stability, especially in deep networks or with high learning rates.

[0052] `--lora_target all`: The llama-factory framework will automatically adjust the placement of LoRA (the adapter) in the model based on different base models (such as Qwen3, Llama3, etc.). When `all` is set, it will attempt to automatically identify and apply it to all linear layers (such as q_proj, v_proj, etc.).

[0053] Optionally, for quantization during the training process: 1. Model loading stage: Use the Linear4bit module of the bitsandbytes library to load the Qwen3-4B model.

[0054] All weights are quantized in 4-bit and stored in video memory.

[0055] 2. Forward Pass: During computation, the 4-bit weights are dynamically dequantized to bf16 for matrix operations.

[0056] This process is real-time and dequantization on demand, so the calculation accuracy is high, but the memory usage is low.

[0057] 3. Gradient Update and LoRA: Only the LoRA adapter parameters (lora_A, lora_B) are trained and updated with full precision (e.g., bf16).

[0058] The main model weights remain frozen and are always 4-bit, and are not updated.

[0059] Therefore, the actual training parameters are limited to the LoRA module, and the backbone model only participates in the forward propagation with "inference-level accuracy".

[0060] The parameters related to the quantization process include: --bf16 True \ --quantization_bit 4 --quantization_method bnb \ --double_quantization True \ The `--bf16 True` option enables 16-bit precision training using Brain Floating Point. Compared to `float16`, it offers a larger dynamic range and reduces the risk of overflow. It also supports gradient accumulation and optimizer state storage, improving training stability.

[0061] The meaning of --quantization_bit 4 is: use 4-bit quantization to compress the underlying model.

[0062] The meaning of `--quantization_method bnb` is: use the bitsandbytes library for quantization.

[0063] --double_quantization True means: Double Quantization First quantization: Convert the weights from float to int8; Second quantization: Perform another quantization on the "scaling constants" (e.g., from float32 to int8). Benefits: Further saves video memory (approximately 0.5~1GB), especially significant when there are a large number of LoRA layers.

[0064] Optionally, the base configuration used during training is as follows: --learning_rate 5e-05 \ --lr_scheduler_type cosine \ --optim adamw_torch \ --max_grad_norm 1.0 --num_train_epochs 1000.0 \ --enable_thinking False \ Here, --learning_rate 5e-05 means that the learning rate for LoRA fine-tuning is set to 5e-05.

[0065] --lr_scheduler_type cosine means: cosine annealing learning rate scheduler, with an initial lr = 5e-5, gradually decreasing to close to 0 (according to the cosine curve), which is more stable than a fixed learning rate and avoids oscillations in the later stages.

[0066] `--optim adamw_torch` indicates that the optimizer is AdamW, a weight decay-corrected version. It uses a native PyTorch implementation, offering good stability.

[0067] The `--max_grad_norm 1.0` option sets the gradient clipping threshold to prevent gradient explosion. If the gradient norm is greater than 1.0, it is scaled down to 1.0.

[0068] --num_train_epochs 1000.0 indicates that the number of training epochs is 1000.

[0069] The `--enable_thinking False` option indicates whether to enable the Chain-of-Thought (CoT) reasoning mode. `False` means the model is not forced to output intermediate reasoning processes.

[0070] When training using multiple CPUs, the relevant CPU allocation is as follows: --flash_attn auto \ --per_device_train_batch_size 2 \ --gradient_accumulation_steps 8 \ flash_attn is a library for accelerating the computation of attention mechanisms. It is widely used in Transformer models, especially when dealing with long sequences, where it can significantly improve training and inference speed and reduce memory usage.

[0071] Flash Attention aims to accelerate attention computation in Transformer models while significantly reducing GPU memory usage. It addresses the issues of excessive GPU memory consumption and potential Out of Memory (OOM) errors caused by long sequences of intermediate matrices, and the slow speed of frequent DRAM read / write operations due to the GPU's limited SRAM capacity for such large data sets, through a "block-based computation + I / O-aware algorithm." Specifically, `--flash_attn auto` enables Flash Attention (if supported) to optimize GPU memory access, accelerating attention computation and reducing memory usage (especially for long sequences), while `auto` automatically detects hardware and CUDA support. `--per_device_train_batch_size2` sets the batch size to 2 per GPU. `--gradient_accumulation_steps 8` specifies the gradient accumulation steps, simulating a larger batch size. Even with limited GPU memory, stable training can still be achieved by simulating larger batches. A "batch" is a core concept in deep learning training; it refers to the set of data samples used each time the model updates its parameters. Batch size is a crucial hyperparameter in deep learning training, determining the number of samples used each time the model updates its parameters. Epoch refers to the number of training iterations, another very important concept in deep learning. One epoch typically means the model has completely "seen" the entire training dataset.

[0072] S103: Perform self-training iterations on the large model according to multiple preset preferences to obtain the target large model.

[0073] Optionally, although the accuracy improved significantly after positive fine-tuning, testing revealed that the large model exhibited some overfitting. One manifestation of this phenomenon was that even with Temperature set to 1, the randomness of the model was essentially eliminated for a certain problem, and this issue was addressed through self-training iterations.

[0074] Optionally, the large model is self-trained iteratively according to multiple preset preferences, including: determining the current large model; constructing a positive and negative preference dataset using the previous generation large model and preferences of the current large model; obtaining golden positive samples based on the positive preference dataset of the previous generation large model; and training the previous generation large model based on the positive and negative preference dataset and the golden positive samples to obtain the current large model.

[0075] Optionally, the positive and negative preference dataset includes key data used to train the model to "distinguish between good and bad." Positive preferences in the dataset refer to outputs that are of higher quality and more in line with human preferences in one or more pairs of responses. Negative preferences refer to outputs that are of lower quality, less desirable, or do not conform to human preferences in the same context.

[0076] Optionally, a positive and negative preference dataset is constructed using the previous generation of the current large model and its preferences. This includes: setting the first parameter of the previous generation large model to a preset value, using the previous generation large model to predict the questions in the second question set multiple times; using a three-order filtering funnel-style multi-preference annotation to label the prediction results of the previous generation large model, and generating a positive and negative preference dataset based on the annotation results. The three-order filtering funnel-style multi-preference annotation includes annotations on preference model accuracy, preference thought chain length, and preference reasoning readability. The first question set and the second question set can be the same or different.

[0077] Optionally, when the contemporary model is a large model fine-tuned under positive supervision, a positive and negative preference dataset can be constructed using the test set from the system optimization prompt word optimization.

[0078] In one embodiment, taking the construction of the positive and negative preference dataset of the Qwen3-4B (Ours_V1) model as an example, the first step is to use the previous generation large model Qwen3-4B (Ours_V0) to predict the questions in the second question set. Five independent predictions are made for each question (or a different number of independent predictions if possible), and the parameter Temperature=1.0 is used during prediction. This results in five different predictions for each question. Next, multi-preference labeling is performed on the five different predictions for each question, ultimately resulting in a positive and negative preference dataset of the Qwen3-4B (Ours_V1) model, which is five times the size of the total question set.

[0079] Optionally, during iterative training, a positive and negative preference dataset is constructed for each generation of the large model, and then the large model is trained using this positive and negative preference dataset.

[0080] Optionally, the first parameter can be Temperature. When obtaining the positive and negative preference dataset, a Temperature=1.0 parameter can be used for the previous generation large model, making the output of the previous generation large model more random and diverse. By using the Temperature=1.0 parameter, the model predicts 5 independent thought chains + answers for each question in each iteration, exploring the diversity of the solution space with a 5 times wider exploration. This can expose the model's weaknesses and also stimulate its hidden potential. The correct answers generated by the model itself are more conducive to the model learning its strengths and avoiding its weaknesses.

[0081] Optionally, a three-stage filtering funnel-style multi-preference annotation can be performed on the five different prediction results for each question to obtain the positive and negative preference dataset of the current generation large model Qwen3-4B(Ours_Vn). Here, the three-stage filtering funnel-style multi-preference annotation actually refers to three preferences: the first preference is model accuracy annotation, indicating that the final answer is correct under a specific format; the second preference is thought chain length annotation, indicating correct prediction data with a length less than the average; and the third preference is reasoning readability annotation, using DeepSeeK-R1 FewShot to assist in judging readability. In this three-stage filtering funnel-style multi-preference annotation, data that passes through the accuracy index, thought chain length index, and readability index in sequence will ultimately be judged as positive preference data.

[0082] Optionally, the golden positive samples are obtained from the positive preference dataset of the previous generation large model, including: obtaining the data corresponding to each question in the positive preference dataset; and filtering the golden positive samples based on the accuracy, length and readability of the data.

[0083] Optionally, the "golden positive sample" can be the most accurate, shortest, and most readable data point for each question in the positive preference dataset. Specifically, the golden positive sample for Qwen3-4B (Ours_V1) is the most accurate, shortest, and most readable data point for each question in the positive preference dataset of Qwen3-4B (Ours_V0). The next generation of golden positive samples, Qwen3-4B (Ours_V2), is the most accurate, shortest, and most readable data point for each question selected by merging the golden positive samples and the positive preference dataset of this generation of Qwen3-4B (Ours_V1). This golden positive sample can prevent forgetting during continuous training and prevent training from failing to converge due to a problem having only negative sample training data.

[0084] Optionally, a previous generation large model is trained based on the positive and negative preference dataset and the golden positive sample to obtain a contemporary large model, including: freezing all parameters of the previous generation large model, training a low-rank adapter of the previous generation large model using the golden positive sample and the positive and negative preference dataset; merging the trained low-rank adapter and the previous generation large model to obtain the contemporary large model.

[0085] In one embodiment, the contemporary large model can be Qwen3-4B (Ours_V1). When training Qwen3-4B (Ours_V1), the parameters of the original model (Qwen3-4B (Ours_V1)) are frozen, and the low-rank adapters (LoRA Adapters) of Qwen3-4B (Ours_V0) are loaded. Using the LLaMA-Factory training framework, the newly obtained training data (golden positive samples, positive and negative preference datasets) are used to fine-tune the low-rank adapters (LoRA Adapters) of Qwen3-4B (Ours_V0) using KTO, resulting in the low-rank adapters (LoRA Adapters) of Qwen3-4B (Ours_V1). Then, the low-rank adapters (LoRA Adapters) of Qwen3-4B (Ours_V1) are merged with the original model to obtain the Qwen3-4B (Ours_V1) model. Each subsequent self-training iteration follows the same pattern: using the previous generation model and data to obtain the training dataset for this generation, and continuously iterating the low-rank adapter based on the parameters of the previous generation model.

[0086] Optionally, each iteration loads the previous version of the adapter as the starting point, serializes and saves the adapter parameters, and improves the accuracy by about 2% per iteration. The three capabilities are continuously improved through repeated iterations. As the number of model iterations increases, the convergence trend of the training loss becomes more and more stable, resulting in a large model that meets the requirements.

[0087] In one embodiment, a comparison test table of the model performance of this application and the model performance of existing solutions is shown in Table 1. The problem-related data used in the test of this solution includes the DISC dataset of financial consulting, financial retrieval, and financial task scenarios, as well as homework data from economics, finance, and accounting textbooks, and financial scenario competition data provided by the AFAC2025 official website. All of these are data from the financial field. During the test, in order to fully evaluate the large model, the parameter Temperature=1.0 was used to make five predictions for each problem in the test set. The test results in Table 1 are the average indicators calculated after prediction.

[0088] In Table 1, the first column, "Model Name," refers to the name of the specific model used during testing; the second column, "System Prompt Words," refers to the preset system prompt words when calling the model; the third column, "Accuracy," is the average of the average accuracy calculated after five predictions for each question in the test set; the fourth column, "Thought Chain Length," is the average of the average thought chain length calculated after five predictions for each question in the test set; the fifth column, "Response Readability," is the average calculated after determining readability for each question in the test set after five predictions. The readability determination uses the DeepSeeK-R1 model and FewShot to construct prompt words to assist in judging the readability of the answer. FewShot (few-shot learning) is a machine learning paradigm whose core goal is to enable the model to quickly adapt to new tasks or identify new categories using a very small number of labeled samples (i.e., "examples"). This application provides examples of negative readability judgments for directly outputting the answer and negative readability judgments for semantically incomprehensible answers in FewShot; the sixth column, "Remarks," is a supplementary explanation of the model and method used during testing. Table 1 clearly shows that while "large-scale inference models" provide the most accurate responses, they still rely on lengthy thought processes and incur significant computational resource costs, making them unsuitable for edge deployment. "Small-parameter, large-scale models," while offering advantages in edge deployment, have limited accuracy, and their lengthy thought processes still impact edge task response efficiency and user experience. System prompts (such as Chain of Draft) simplify the thought process, but still suffer from reduced readability and lower accuracy.

[0089] This application optimizes the model inference path while ensuring accuracy, efficiently compressing redundant content to reduce resource costs and improve execution efficiency. It primarily targets edge deployment and small-parameter model scenarios, achieving high-efficiency inference with low latency and low power consumption. By balancing the preferences for accuracy, readability, and thought chain length, the final model of this solution achieves the impressive performance shown in Table 1. The large model ultimately obtained in this application possesses practical and intelligent capabilities for high real-time and high-security scenarios in the financial field, as well as feasibility for implementation.

[0090] Table 1

[0091] In one alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0092] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0093] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not indicate that there is only one bus or one type of bus.

[0094] The memory 4003 may be ROM (Read-Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read-Only Memory), CD-ROM (Compact Disc Read-Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0095] The memory 4003 stores computer programs that execute embodiments of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0096] Among them, electronic devices can be any electronic product that can interact with an object, such as personal computers, tablets, smartphones, personal digital assistants (PDAs), game consoles, interactive network television (IPTV), smart wearable devices, etc.

[0097] The electronic device may also include network devices and / or object devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers for cloud computing.

[0098] The networks in which the electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).

[0099] The terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the illustrations or text descriptions.

[0100] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0101] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A large-scale model optimization method based on multi-preference alignment and thought chain compression, characterized in that, The method includes: The system prompt words of the large model are optimized based on the length of the thought chain and the accuracy of the questions in the test set. A training dataset is constructed based on the optimized system prompts, and the large model is then fine-tuned using the training dataset. The large model is self-trained iteratively based on multiple preset preferences to obtain the target large model.

2. The large model optimization method based on multi-preference alignment and thought chain compression according to claim 1, characterized in that, The system prompts based on the large model optimized for the length of the thought chain and the accuracy of questions in the test set include: Determine a large model to be trained, and set the first hyperparameter of the large model to a preset value, wherein the first hyperparameter includes Temperature. The large model is used to predict the problems in the test set multiple times, and the accuracy and corresponding thought chain length are obtained based on the prediction results. The system prompts are adjusted based on the accuracy rate and the length of the thought chain.

3. The large model optimization method based on multi-preference alignment and thought chain compression according to claim 1, characterized in that, The step of constructing a training dataset based on the optimized system prompts includes: The first question set is labeled using multiple labeling models to obtain the labeling results, which include the initial answers corresponding to the first question set. The target answer for the first question set is determined based on the annotation results, and the training dataset is constructed based on the target answer, the first question set, and the optimized system prompts.

4. The large model optimization method based on multi-preference alignment and thought chain compression according to claim 3, characterized in that, Determining the target answer for the first question set based on the annotation results includes: Obtain all initial answers for each question in the first question set; The target answer for each question is determined based on the conditions corresponding to the initial answer and the preset voting method, wherein the preset voting method includes majority voting and weighted voting.

5. The large model optimization method based on multi-preference alignment and thought chain compression according to claim 1, characterized in that, The large model is positively fine-tuned using the training dataset, including: Freeze all parameters of the large model and use the training dataset to perform KTO training on the low-rank adapter corresponding to the large model; The trained low-rank adapter is merged with the large model to obtain a new large model.

6. The large model optimization method based on multi-preference alignment and thought chain compression according to claim 2, characterized in that, The step of performing self-training iterations on the large model based on multiple preset preferences includes: Identify the current large model to be acquired, and construct a positive and negative preference dataset using the previous generation large model of the current large model and the preferences; Obtain the gold positive sample based on the positive preference dataset of the previous generation large model; The previous generation large model is trained based on the positive and negative preference dataset and the golden positive sample to obtain the contemporary large model.

7. The large model optimization method based on multi-preference alignment and thought chain compression according to claim 6, characterized in that, The construction of a positive and negative preference dataset using the previous generation of the current large model and the preferences includes: Set the first parameter of the previous generation large model to a preset value, and use the previous generation large model to predict the problems in the second problem set multiple times. The three-order filtering funnel-style multi-preference annotation uses the prediction results of the previous generation large model to generate the positive and negative preference dataset based on the annotation results. The three-order filtering funnel-style multi-preference annotation includes preference model accuracy annotation, preference thought chain length annotation, and preference reasoning readability annotation.

8. The large model optimization method based on multi-preference alignment and thought chain compression according to claim 6, characterized in that, The process of obtaining the golden positive samples based on the positive preference dataset of the previous generation large model includes: Obtain the data corresponding to each question in the positive preference dataset; The gold standard positive samples are selected based on the accuracy, length, and readability of the data.

9. The large model optimization method based on multi-preference alignment and thought chain compression according to claim 6, characterized in that, The process of training the previous generation large model based on the positive and negative preference dataset and the golden positive sample to obtain the current generation large model includes: Freeze all parameters of the previous generation large model, and train a low-rank adapter of the previous generation large model using the golden positive samples and the positive and negative preference dataset; The trained low-rank adapter and the previous generation large model are merged to obtain the current generation large model.

10. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 9.

Images