Domain-specific large model oriented multi-granularity preference signal enhanced alignment training method

Abstract

This application provides a multi-granularity preference signal enhancement and alignment training method for large-scale models in a specific domain. The method includes: first, constructing a domain-specific preference training dataset containing expert quality scores and task background factor annotations; then, semantically segmenting the long-chain inference response and implementing segment-level differential optimization, fusing score gap information to increase preference signal density, and eliminating background clutter interference through a causal backdoor adjustment method; finally, weighted combining the three types of losses into a total joint loss, and iteratively optimizing the model parameters to complete preference alignment. This application can effectively improve the model's long-chain inference capability, training convergence efficiency, and cross-scenario generalization ability, while being compatible with existing training infrastructure and having low engineering implementation costs.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a multi-granularity preference signal enhancement alignment training method for large models in a specific domain. Background Technology

[0002] After being pre-trained on large-scale general corpora, large language models possess rich knowledge reserves and natural language generation capabilities. However, when applied in vertical professional fields, their output often deviates significantly from the business standards and decision-making expectations of domain experts, making it difficult to directly meet the high-quality application needs of professional scenarios. To ensure that the generation behavior of large models aligns with human intentions and domain norms, the industry has successively developed several preference alignment techniques: Traditional Human Feedback Reinforcement Learning (RLHF) trains reward models independently to fit human preferences and then iteratively optimizes the generation strategy using reinforcement learning. While it achieves a certain alignment effect, the overall technical chain is long, the engineering complexity is high, the training process is unstable, and the implementation threshold is high. The subsequent emergence of Direct Preference Optimization (DPO) simplifies preference learning into a supervised classification problem of paired samples. It can directly complete model alignment without explicitly building a reward model, significantly reducing engineering implementation costs and training complexity, and has become the mainstream technical path for preference alignment of large models.

[0003] In professional decision support scenarios with stringent requirements for long-chain inference depth, output reliability, and cross-scenario adaptability, existing direct preference optimization methods still have significant technical shortcomings. The core issue lies in the insufficient overall quality of preference signals, which is specifically reflected in three dimensions. First, the preference signal granularity is coarse. Existing methods treat the complete model response as a single optimization unit and train it with a uniform regularization strength. However, long-chain inference responses in professional fields typically encompass two semantically significantly different components: factual descriptions and logical deductions. The learning difficulty of these two components differs greatly, and a uniform optimization mode can easily lead to an imbalance, such as overfitting of simple descriptive content and insufficient training of core reasoning capabilities. Second, the preference signal density is low. Existing methods only utilize binary ranking information of "which is better" and do not explore the degree of quality difference between paired responses. The sparse signal cannot guide the model to match differentiated learning strengths, resulting in low training convergence efficiency and a high proportion of invalid parameter updates. Third, the preference signal purity is insufficient. The generation process of preference data in professional fields is affected by various background factors such as task type, business level, and application scenario. These factors can introduce spurious correlations into the data that are unrelated to the true quality of the response. Existing methods directly fit the observed preference relationships, causing the model to learn background bias rather than the true quality evaluation criteria. Ultimately, this results in weak cross-scenario generalization ability and insufficient output fairness, failing to fully meet the high-standard application requirements of professional fields. Summary of the Invention

[0004] In view of this, embodiments of this application provide a multi-granularity preference signal enhancement and alignment training method, apparatus, electronic device and storage medium for large models in specific domains. Through systematic improvements to the preference signals by refining the granularity, increasing the density and enhancing the purity, it can significantly improve the long-chain inference capability, training convergence efficiency and cross-scenario generalization robustness of large models in professional domains while being compatible with existing training infrastructure, and has low engineering implementation costs.

[0005] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide a multi-granularity preference signal enhancement and alignment training method for large models in a specific domain, comprising the following steps: Construct a domain-specific preference training dataset, in which each sample contains domain task prompts, pairs of preferred and rejected responses, expert quality scores, and annotations of task context factors; Semantic segmentation is performed on each pair of candidate responses, and the long-chain inference response of a specific domain is split into description segment and inference segment. Dynamic optimization weights are calculated based on segment-level score differences, segment-specific KL regularization parameters are mapped and generated, and segment-level joint loss is constructed. The expert rating difference between paired responses is extracted, and the rating gap is incorporated into the optimization objective as a marginal correction term for preference. A direct preference optimization loss based on rating perception is constructed, and the binary ranking signal is transformed into a continuous intensity signal. Based on the causal inference backdoor adjustment method, the background factors that affect preference labeling are marginalized, the spurious correlation backdoor path introduced by the background factors is cut off, and a causal deconfounding loss is constructed. The total joint loss is obtained by weighting and summing the segment-level joint loss, rating perception loss, and causal decontamination loss according to a preset balance coefficient. Based on the total joint loss, the parameters of the large model in a specific domain are iteratively optimized to complete the preference alignment training.

[0006] Secondly, embodiments of this application also provide a multi-granularity preference signal enhancement and alignment training device for large models in a specific domain, the device comprising: The building module is used to construct a domain-specific preference training dataset. Each sample contains domain task hints, pairs of preferred and rejected responses, expert quality scores, and annotations of task context factors. Semantic segmentation is performed on each pair of candidate responses, and the long-chain inference response of a specific domain is split into description segment and inference segment. Dynamic optimization weights are calculated based on segment-level score differences, segment-specific KL regularization parameters are mapped and generated, and segment-level joint loss is constructed. The expert rating difference between paired responses is extracted, and the rating gap is incorporated into the optimization objective as a marginal correction term for preference. A direct preference optimization loss based on rating perception is constructed, and the binary ranking signal is transformed into a continuous intensity signal. Based on the causal inference backdoor adjustment method, the background factors that affect preference labeling are marginalized, the spurious correlation backdoor path introduced by the background factors is cut off, and a causal deconfounding loss is constructed. The total joint loss is obtained by weighting and summing the segment-level joint loss, rating perception loss, and causal decontamination loss according to a preset balance coefficient. Based on the total joint loss, the parameters of the large model in a specific domain are iteratively optimized to complete the preference alignment training.

[0007] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the multi-granularity preference signal enhancement alignment training method for a large model in a specific domain as described in any of the first aspects.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the multi-granularity preference signal enhancement alignment training method for a large model in a specific domain as described in any one of the first aspects.

[0009] The embodiments of this application have the following beneficial effects: By responding to semantic segmentation and segment-level differentiated optimization mechanisms, appropriate optimization intensities can be assigned to semantic components with different functions and learning difficulties in long-chain inference responses. This ensures that core logical reasoning capabilities are fully trained and effectively improves the imbalance problem of overfitting simple content and under-optimizing core reasoning content under uniform optimization intensity. By integrating expert rating gap information to increase the density of preference signals, sparse binary ranking signals are transformed into continuous intensity signals. This allows the model to adaptively adjust the parameter update intensity based on the quality difference of paired responses, significantly reducing invalid gradient updates, effectively accelerating training convergence speed, and reducing overall training computational overhead. By eliminating pseudo-correlation interference in the data through a background factor causal decontamination mechanism, the alignment strategy learned by the model truly reflects the quality level of the response itself, significantly improving the model's cross-scenario generalization ability and output fairness across different business scenarios and task types. At the same time, the overall solution is a lightweight modification based on the existing direct preference optimization framework, without modifying the network structure and inference logic of large models. It can be seamlessly adapted to mainstream large model training infrastructure, with low engineering implementation costs and strong scenario adaptability. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating steps S101-S105 provided in the embodiments of this application; Figure 2 This is a flowchart illustrating steps S201-S205 provided in the embodiments of this application; Figure 3 This is a flowchart illustrating steps S301-S302 provided in the embodiments of this application; Figure 4 This is a flowchart illustrating steps S401-S404 provided in the embodiments of this application; Figure 5 This is a flowchart illustrating steps S501-S503 provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the multi-granularity preference signal enhancement and alignment training device for large models in a specific domain provided in this application embodiment; Figure 7 This is a schematic diagram of the composition structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0013] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0014] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0015] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0016] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application and is not intended to limit the scope of this application.

[0018] See Figure 1 , Figure 1 This is a flowchart illustrating steps S101-S105 of the multi-granularity preference signal enhancement and alignment training method for large models in a specific domain provided in this application embodiment. It will be combined with... Figure 1 Steps S101-S105 are explained below.

[0019] In step S101, a domain-specific preference training dataset is constructed, in which each sample contains domain task prompts, pairs of preferred and rejected responses, expert quality score data, and task background factor annotations. In step S102, semantic segmentation is performed on each pair of candidate responses, and the long-chain inference response of a specific domain is split into a description segment and an inference segment. Dynamic optimization weights are calculated based on the segment-level score differences, segment-specific KL regularization parameters are mapped and generated, and segment-level joint loss is constructed. In step S103, the expert rating difference between paired responses is extracted, and the rating difference is incorporated into the optimization objective as a preference marginal correction term to construct the direct preference optimization loss based on rating perception, thereby converting the binary ranking signal into a continuous intensity signal. In step S104, based on the causal inference backdoor adjustment method, the background factors affecting preference labeling are marginalized, the pseudo-correlation backdoor path introduced by the background factors is cut off, and the causal deconfounding loss is constructed. In step S105, the segment-level joint loss, rating perception loss, and causal decontamination loss are weighted and summed according to a preset balance coefficient to obtain the total joint loss. Based on the total joint loss, the parameters of the large model in a specific domain are iteratively optimized to complete the preference alignment training.

[0020] This embodiment first constructs a domain-specific preference training dataset. The training samples do not only contain basic data such as prompts and paired preference labels, but also integrate multi-dimensional annotation information such as expert quality scores, task background factors, and semantic segmentation criteria. This provides corresponding data input support for the subsequent three stages of segment-level differentiation optimization, score gap fusion, and causal decontamination, ensuring that each enhancement module has matching labeled data as the basis for calculation, without the need to generate or infer relevant features during the training process.

[0021] To address the semantic differences in long-chain reasoning responses across specialized domains, this embodiment decomposes the complete response into two semantic components: a descriptive segment stating objective facts and a reasoning segment executing logical deductions. Dynamically optimized weights are calculated based on the quality differences between these two segments, and further mapped to differentiated regularization parameters. This achieves differentiated training intensity allocation for content of varying learning difficulty. Its core function is to overcome the imbalance in content learning caused by uniform optimization intensity, ensuring that the model fully optimizes high-difficulty core reasoning capabilities while avoiding overfitting in the low-difficulty factual description portion.

[0022] Then, the quality gap information contained in the expert ratings is extracted, transforming the sparse ranking signal into a continuous intensity signal that reflects the degree of excellence. During training and optimization, the model can adaptively adjust the magnitude of parameter updates based on the quality difference between paired responses: samples with large quality gaps output stronger gradient update signals, and the model increases its learning intensity; samples with small quality gaps output smoother gradient update signals, and the model learns moderately by referencing these signals, thereby reducing invalid parameter updates and improving training convergence efficiency.

[0023] Then, based on causal inference theory, a structured model of the preference data generation process is constructed. This model identifies backdoor paths formed by background factors simultaneously influencing cue construction, response generation, and preference labeling. These paths introduce spurious correlations into the data that are unrelated to the true quality of the response. By employing a backdoor adjustment method, the prior distribution of background factors is marginalized, cutting off the spurious correlations introduced by confounding paths. This ensures that the alignment strategy learned by the model is determined solely by the quality of the response itself, rather than by biases from background factors such as business scenarios and hierarchical requirements. This fundamentally improves the model's generalization ability and fairness across different background scenarios.

[0024] Finally, the three enhancement modules are integrated into a unified optimization system. The loss functions of the three modules are weighted and combined according to a preset balance coefficient to form a unified total loss function. In each training iteration, the three types of losses are calculated simultaneously, backpropagation gradients are merged, and model parameters are updated to achieve the synergistic superposition of the three effects of granular refinement, density enhancement, and purity enhancement. This avoids the effect cancellation or training cost increase caused by staged training, and ultimately completes the full-dimensional preference alignment training of the large model.

[0025] This embodiment boasts high compatibility and practicality. The overall training process is fully compatible with existing direct preference optimization training pipelines and can seamlessly integrate with mainstream large-scale model training frameworks and distributed training tools. During engineering implementation, only segmentation processing and annotation parsing logic need to be added to the data preprocessing module, and the calculation logic for three types of augmentation losses needs to be added to the loss calculation module. No modifications to the main model code or inference logic are required, resulting in low engineering transformation costs. It can be quickly migrated to large-scale model alignment training tasks with different parameter scales and professional fields.

[0026] In some embodiments, see Figure 2 , Figure 2 This is a flowchart illustrating steps S201-S205 provided in the embodiments of this application. The step of semantically segmenting each pair of candidate responses, splitting the long-chain inference response of a specific domain into a description segment and an inference segment, calculating dynamically optimized weights based on segment-level score differences, mapping and generating segment-specific KL regularization parameters, and constructing segment-level joint loss can be achieved through steps S201-S205, which will be explained in conjunction with each step.

[0027] In step S201, each long chain reasoning response is automatically divided into a description segment for stating objective facts about scene elements, domain object attributes and parameters, and a reasoning segment for performing scheme deduction and multi-hop logic analysis, by using predefined structured tags or rule segmenters. In step S202, based on the domain-specific comprehensive scoring function, the quality scores of the description segment and the reasoning segment in the preferred response and the rejection response are calculated respectively, and the preference difference ΔR_desc of the description segment and the preference difference ΔR_reason of the reasoning segment are obtained. In step S203, a temperature parameter τ is introduced, and the preference difference between the two segments is normalized by the softmax function to calculate the dynamic optimization weight w_s corresponding to the description segment and the inference segment. In step S204, the dynamic weights are mapped to segment-specific KL regularization parameters β_s based on the basic KL regularization parameter β and combined with the scaling factor α. In step S205, based on the segment-specific KL regularization parameters of the two segments, the preference margins of the description segment and the inference segment are weighted and combined to construct the segment-level joint loss function L_segment.

[0028] This embodiment first performs response semantic segmentation. Using predefined structured markers or a segmenter based on domain semantic rules, each long-chain inference response is automatically split into a description segment and an inference segment. The description segment contains objective facts such as scene elements, domain object attributes, and device parameters, with the core requirement being factual accuracy and complete information. The inference segment contains content related to scheme deduction, multi-hop logic analysis, and causal relationship judgment, with the core requirement being logical rigor and reasonable deduction. The difference in semantic function between the two segments determines the different learning difficulties of their models, thus requiring different optimization intensities.

[0029] After segmentation, call the domain-specific comprehensive scoring function score( ), calculate the quality scores of each paragraph for the preferred response and the rejection response respectively, and then obtain the preference difference between the two paragraphs. The calculation formula is: ΔR_desc=score(R + _desc)-score(R - _desc); ΔR_reason=score(R + _reason)-score(R - _reason); Where R + _desc、R + _reason represents the description section and reasoning section of the preferred response, respectively. - _desc、R - _reason represents the description segment and the inference segment for rejecting the response, respectively. Based on this, a temperature parameter τ is introduced, and the preference difference between the two segments is normalized using the softmax function to calculate the dynamic optimization weights of each segment: w_s=exp(τ ΔR_s) / [exp(τ ΔR_desc)+exp(τ ΔR_reason)],s∈{desc,reason}; The temperature parameter τ is used to adjust the discrimination sensitivity of the weights. The larger the value of τ, the more significant the influence of the preference difference on the weights.

[0030] After obtaining the dynamic weights, using the basic KL regularization parameter β as a benchmark, and combining it with the scaling factor α, the weights are mapped to segment-specific KL regularization parameters. The calculation formula is as follows: β_s=β×[1+α(2w_s-1)],s∈{desc,reason}; Where 0 < α < 1, this controls the magnitude of parameter adjustment. This mapping logic achieves adaptive allocation of optimization strength: paragraphs with weights higher than the average correspond to larger regularization parameters, resulting in stronger optimization; paragraphs with weights lower than the average correspond to smaller regularization parameters, resulting in weaker optimization. Finally, based on the segment-specific KL regularization parameters of the two segments, the preference margins F_s of the description and inference segments are weighted and combined to construct a segment-level joint loss function: L_segment=-logσ(β_desc F_desc+β_reason F_reason); Where σ( F_s is the sigmoid function, which measures the ability of the policy model to distinguish between positive and negative examples in the corresponding paragraph relative to the reference model.

[0031] In some embodiments, see Figure 3 , Figure 3 This is a flowchart illustrating steps S301-S302 provided in the embodiments of this application. The step of extracting the expert rating difference between paired responses, incorporating the rating difference as a preference marginal correction term into the optimization objective, constructing a rating-perceived direct preference optimization loss, and converting the binary ranking signal into a continuous intensity signal can be achieved through steps S301-S303, which will be explained in conjunction with each step.

[0032] In step S301, the quality scores of the preferred responses and the rejection responses are extracted from the domain expert annotation data, and the score difference Δ between the two is calculated. ; In step S302, a rating confidence parameter γ is introduced, and the rating gap is used as a correction term for the standard preference margin. A rating-aware direct preference optimization loss function L_rating is constructed, and the learning signal strength is adaptively adjusted according to the size of the rating gap. In step S303, for training samples without rating labels, the loss function is automatically degenerated into the standard direct preference optimization loss form.

[0033] This embodiment first extracts the scoring gap data, and from the annotation results of domain experts, obtains the domain task hint x and the preferred response a corresponding to each sample. + Refuse to respond to a - And the quality scores given by experts for the two responses. (x,a + )and (x,a -The score difference for this sample is obtained by calculating the difference between the two scores. The calculation formula is as follows: Δ (x,a + ,a - )= (x,a + )- (x,a - ); The magnitude of the score difference directly reflects the degree of quality difference between paired responses. The larger the difference, the more obvious the quality advantage of the preferred response relative to the rejection response, and the stronger the learning guidance signal.

[0034] After extracting the rating gap, a direct preference optimization loss function based on rating perception is constructed. A rating trust parameter γ (γ>0) is introduced to adjust the weight of the impact of rating gap information on the optimization objective; the parameter value can be flexibly adjusted based on the professional reliability of expert ratings. The rating gap is used as a correction term for the standard preference margin, and the loss calculation formula for the batch dimension is as follows: L_rating=Σ i -logσ(β Δ i _θ-γ Δ i _ ); Where N is the number of samples in the training batch, β is the KL regularization parameter, and Δ i _θ represents the log-likelihood ratio difference of the policy model for the i-th sample, and Δ i _ Let be the score difference of the i-th sample. The operating logic of this loss function is as follows: when the score difference is large, the correction term will increase the effective value of the preference margin, corresponding to a larger gradient update signal, and the model will increase its learning strength; when the score difference is small, the correction term will decrease the effective value of the preference margin, corresponding to a smoother gradient update signal, and the model will learn with moderate reference.

[0035] To improve the method's scenario compatibility and noise robustness, two supplementary mechanisms are implemented. The first is an automatic degradation mechanism: for training samples without rating labels, the rating difference Δ is... Setting it to 0 automatically degenerates the loss function into the standard direct preference optimization loss form, which is directly compatible with datasets of mixed labeled quality without requiring additional data splitting. The second term is the maximum likelihood extension form, which, for scenarios with high rating noise, can use a loss function based on the maximum likelihood principle: L_ML=Σ i [-logσ(β Δ i _θ)+(Δ i _ -β Δ i _θ)² / (2V)]; Where V is the variance parameter of the score difference (V>0), reflecting the confidence in the score information. The squared regularization term can constrain the impact of score deviation on the optimization objective and further improve the robustness to score noise. It can be selected and used according to the actual data quality.

[0036] In some embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating steps S401-S404 provided in the embodiments of this application. The method based on causal inference backdoor adjustment marginalizes the background factors that influence preference labeling, cuts off the pseudo-correlation backdoor path introduced by the background factors, and constructs a causal deconfounding loss. This can be achieved through steps S401-S404, which will be explained in conjunction with each step.

[0037] In step S401, the process of generating preference data is modeled as a causal structure containing background factor U, and the background factor is identified to act on the backdoor path formed by prompt construction, response generation and preference labeling simultaneously. In step S402, the backdoor adjustment method in causal inference is adopted to perform marginalization calculation based on the prior distribution p(U) of background factors, and cut off the backdoor path to eliminate background confounding bias. In step S403, during the training process, K background factor instances are sampled from the prior distribution of background factors for each training batch, each instance is embedded into the conditional position of the prompt, and the log-likelihood probability of the policy model and the reference model is calculated respectively. In step S404, the calculation results of K sampled instances are marginalized and averaged to construct the causal decontamination loss function L_causal.

[0038] This embodiment first constructs a causal structure model of preference data, abstracting the generation process of preference data in a specific domain into a causal graph containing background factor U. The causal relationship between the variables is as follows: background factor U influences the construction of domain cue X, cue X further influences the generation of candidate responses Y1 and Y2, and finally, the quality of the candidate responses and the background factor jointly determine the labeling result of preference label L. The corresponding causal path can be represented as: U→X→(Y1,Y2)→L; U→L; The path where background factor U acts on both prompt generation and preference labeling is a backdoor path. This path introduces pseudo-correlation into the data that is unrelated to the true quality, causing the model to learn background bias rather than the true quality evaluation criteria.

[0039] To eliminate the confounding effects of backdoor paths, a backdoor adjustment method from causal inference is employed. This involves marginalizing the prior distribution p(U) of background factors to obtain the true preference distribution after eliminating background bias. The calculation formula is as follows: p(L|do( ))=Σ u p(L|X,Y1,Y2,U=u) p(U=u); This calculation smooths out the preference bias caused by specific background values ​​by weighting the values ​​of all possible background factors, so that the optimization objective focuses on the quality difference of the response itself, and ensures that the learned strategy has the ability to generalize across background scenarios.

[0040] During training, decongestion calculations are performed in batches. The specific steps are as follows: First, for each training batch, K background factor instances u are sampled from the pre-statistically obtained prior distribution of background factors. k The typical value of K is 5~20, which can be adjusted according to the number of dimensions of the background factors and the data scale. Then, each background factor instance is embedded into the conditional position of the prompt, constructing a prompt input with background conditions. Next, the log-likelihood probability of the policy model and the reference model for the candidate response under each conditional prompt is calculated, deriving the preference margin for each background instance. Finally, the calculation results of the K sampled instances are marginalized and averaged to construct the causal deconfounding loss function. L_causal=-E[(1 / K)Σ k logσ(β Δ_θ(x,u k ))]; Where Δ_θ(x,u) k ) is the background instance u k The log-likelihood ratio difference of the lower strategy model, where β is the KL regularization parameter, and σ( ) is the sigmoid function.

[0041] In some embodiments, the expression for the total joint loss function is: L_total = L_segment + λ1 L_rating+λ2 L_causal; Where L_segment is the segment-level joint loss function, L_rating is the rating-aware direct preference optimization loss function, L_causal is the causal deconfounding loss function, and λ1 and λ2 are module balancing coefficients, typically taking values ​​of 0.5 and 0.6 respectively, which are adjusted based on the convergence of the validation set. Training uses the AdamW optimizer with a learning rate set to 1×10⁻⁶. -6 Up to 5×10-6 Each batch contains 32-128 pairs of preference samples, and the training rounds are 3-5 epochs. The training process is combined with a learning rate scheduling strategy of linear warm-up and cosine annealing.

[0042] The loss functions of the three modules are weighted and combined using module balancing coefficients to form the total joint loss function, calculated as follows: L_total = L_segment + λ1 L_rating+λ2 L_causal; Wherein λ1 is the balancing coefficient of the score gap fusion module, typically set to 0.5, used to adjust the weight of the score intensity signal in the overall optimization objective; λ2 is the balancing coefficient of the causal decontamination module, typically set to 0.6, used to adjust the weight of decontamination optimization in the overall objective. These two coefficients are not fixed values ​​and can be fine-tuned based on the convergence of the validation set and the evaluation results of downstream tasks to adapt to training scenarios with different domains, data scales, and alignment requirements.

[0043] Training employs the AdamW optimizer for parameter updates. This optimizer introduces a weight decay mechanism on top of gradient descent, effectively reducing the risk of model overfitting and improving training stability. The learning rate is set to 1×10⁻⁶. -6 Up to 5×10 -6 In the training interval, preference alignment is part of the model fine-tuning stage. A lower learning rate can prevent catastrophic forgetting in the model and ensure a smooth and stable alignment process. Each batch of training contains 32-128 pairs of preference samples, which can be flexibly adjusted within this range according to GPU memory capacity and training efficiency requirements. Larger batches can improve the stability of gradient estimation, while smaller batches can reduce hardware resource requirements.

[0044] The overall training epochs are set to 3-5 epochs to ensure the model fully learns the preference signals while avoiding overtraining that could lead to a decline in general capabilities. During training, a learning rate scheduling strategy combining linear warm-up and cosine annealing is employed: a linear warm-up phase is set in the early stages of training, with the learning rate gradually increasing from 0 to a set value to prevent excessive gradient fluctuations in the initial stage that could cause model parameter instability; in the later stages of training, a cosine annealing strategy is used, with the learning rate gradually and smoothly decreasing from its peak value, allowing the model parameters to gradually converge to the optimal solution, further improving the convergence effect and the final alignment quality.

[0045] In some embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating steps S501-S503 provided in the embodiments of this application. The construction of a specific domain preference training dataset can be achieved through steps S501-S503, which will be explained in conjunction with each step.

[0046] In step S501, domain prompt data covering multiple typical business scenarios is collected, and the background factors corresponding to each prompt, such as task type, business attributes, hierarchical requirements, and security level requirements, are labeled. In step S502, a baseline large model, which has been fine-tuned under supervision by a domain-specific professional corpus, is used to sample and generate multiple candidate responses for each prompt, and a preference pair candidate pool is constructed. In step S503, domain experts conduct multi-dimensional quality assessments of candidate responses and provide sub-item scores. Based on the comprehensive scores, pairwise preference ranking labels, overall score gaps, and segment-level score data are generated.

[0047] First, domain suggestion data covering multiple typical business scenarios is collected. Suggestion sources can include anonymized data from real business scenarios, synthetic scenario-generated data, and historical business archive data, ensuring comprehensive task coverage, balanced scenario distribution, and representation of various query needs in actual business. Each suggestion is annotated with corresponding multi-dimensional background factor information, including task type, business attributes, hierarchical requirements, and confidentiality requirements. This serves two purposes: firstly, to construct the prior distribution of background factors, and secondly, to provide a basis for conditional sampling for the causal decontamination module.

[0048] Next, a baseline large model, fine-tuned under supervision with a domain-specific professional corpus, was used as the response generation model. For each prompt, multiple candidate responses were generated through random sampling to construct a preference pair candidate pool. During the sampling process, generation parameters such as temperature, kernel sampling threshold, and penalty coefficient were adjusted to obtain candidate responses of different qualities, inference paths, and expression styles. This ensured that the responses in the candidate pool had sufficient quality differences and content diversity, providing a sufficient basis for comparison in subsequent expert annotation and preference pair construction.

[0049] Finally, experts in the relevant fields conducted a multi-dimensional quality assessment of the candidate responses, providing specific sub-scores for each dimension, including factual accuracy, logical rigor, and content completeness. A comprehensive quality score was generated based on a weighted average of these sub-scores. This score was then used to produce three types of annotation results: the first type is pairwise preference ranking labels, where candidate responses to the same prompt are compared pairwise to select preferred and rejected responses, forming binary preference pairs; the second type is the overall score difference, the difference in comprehensive scores between each pair of preference responses, used in the score difference fusion module; and the third type is segment-level scoring data, where independent quality scores are given to the description and reasoning parts of the response, used in the segment-level differentiation optimization module.

[0050] In some embodiments, the trained alignment model is directly deployed in a specific domain decision support scenario. After receiving domain business queries in text form, it outputs a structured reasoning report containing scenario description, object identification, risk assessment, and action suggestions.

[0051] The trained alignment model can be directly deployed in domain-specific decision support scenarios, providing intelligent assistance services to business personnel. After receiving domain-specific business queries in text form from users, the model automatically outputs a structured inference report through inference computation. The report includes core modules such as scenario description, object identification, risk assessment, and action suggestions, which precisely correspond to the semantic division of the description and inference sections in the training phase. This ensures both the accurate presentation of objective factual information and the rigor and rationality of the logical reasoning process, meeting the structured requirements of professional domain scenarios for output content.

[0052] This model boasts exceptional ease of deployment. Since this method only improves the loss function design and data processing flow during the training phase, without modifying the network structure, parameter forms, or inference logic of the large model, no adaptation or modification to existing large model inference frameworks is required during deployment. It can be directly integrated into existing model service infrastructure. Furthermore, the model does not require any additional alignment post-processing steps during the inference phase, maintaining inference latency consistent with the baseline fine-tuned model. This meets the timeliness requirements of professional domain scenarios and can be quickly deployed to various real-time decision support scenarios.

[0053] In summary, the embodiments of this application have the following beneficial effects: By implementing multi-dimensional systematic improvements to the preference signal, including granular refinement, density enhancement, and purity improvement, high-quality preference alignment for large-scale professional domain models is achieved within the direct preference optimization framework. By splitting long-chain inference responses into description and inference segments and implementing segment-level differentiated optimization, the learning intensity of different semantic components can be adaptively matched, effectively avoiding the imbalance problem of overfitting simple content and under-optimization of core inference content, significantly improving the quality of long-chain logical inference. By fusing expert rating gap information to transform sparse binary ranking signals into continuous intensity signals, the model can precisely adjust the learning intensity based on response quality differences, reducing invalid parameter updates and effectively accelerating training convergence and improving training efficiency. A backdoor adjustment method based on causal inference eliminates spurious correlation interference from background factors, ensuring that the preference strategies learned by the model truly reflect the quality level of the content itself, significantly improving the model's generalization ability and output fairness across different business scenarios. Simultaneously, the overall solution is fully compatible with existing large-scale model training infrastructure, requiring no modification to the model network architecture, resulting in low engineering implementation costs and convenient deployment.

[0054] Based on the same inventive concept, this application also provides a multi-granularity preference signal enhancement and alignment training device for a specific domain large model, which corresponds to the multi-granularity preference signal enhancement and alignment training method for a specific domain large model in the first embodiment. Since the principle of the device in this application is similar to the above-mentioned multi-granularity preference signal enhancement and alignment training method for a specific domain large model, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0055] like Figure 6 As shown, Figure 6 This is a schematic diagram of the structure of a multi-granularity preference signal enhancement and alignment training device 600 for a large-scale model in a specific domain, provided in an embodiment of this application. The multi-granularity preference signal enhancement and alignment training device 600 for a large-scale model in a specific domain includes: The first construction module 601 is used to construct a domain-specific preference training dataset. Each sample contains domain task prompts, pairs of preferred and rejected responses, expert quality score data, and task background factor annotations. The second construction module 602 is used to perform semantic segmentation on each pair of candidate responses, split the long chain inference response of a specific domain into description segments and inference segments, calculate dynamic optimization weights based on segment-level score differences, map and generate segment-specific KL regularization parameters, and construct segment-level joint loss. The third construction module 603 is used to extract the expert rating difference between paired responses, incorporate the rating gap as a preference marginal correction term into the optimization objective, construct the rating-perceived direct preference optimization loss, and transform the binary ranking signal into a continuous intensity signal. The fourth construction module 604 is used to marginalize the background factors that affect preference labeling based on the causal inference backdoor adjustment method, cut off the spurious correlation backdoor path introduced by the background factors, and construct causal deconfounding loss. The iterative optimization module 605 is used to sum the segment-level joint loss, rating perception loss and causal decontamination loss by weighting them according to a preset balance coefficient to obtain the total joint loss. Based on the total joint loss, the parameters of the large model in a specific domain are iteratively optimized to complete the preference alignment training.

[0056] Those skilled in the art should understand that Figure 6 The implementation functions of each unit in the multi-granularity preference signal enhancement and alignment training device 600 for a large model for a specific domain shown can be understood by referring to the relevant description of the multi-granularity preference signal enhancement and alignment training method for a large model for a specific domain. Figure 6 The functions of each unit in the multi-granularity preference signal enhancement and alignment training device 600 for a large model in a specific domain can be implemented by a program running on a processor or by specific logic circuits.

[0057] In one possible implementation, the second building module 602 is specifically used for: By using predefined structured tags or rule segmenters, each long chain reasoning response is automatically divided into a descriptive segment that presents objective facts about scene elements, domain object attributes and parameters, and a reasoning segment that is used for execution scheme deduction and multi-hop logic analysis. Based on a domain-specific comprehensive scoring function, the quality scores of the description segment and the reasoning segment in the preferred response and the rejection response are calculated respectively, and the preference difference ΔR_desc of the description segment and the preference difference ΔR_reason of the reasoning segment are obtained. Introducing the temperature parameter τ, the preference difference between the two segments is normalized using the softmax function, and the dynamic optimization weight w_s corresponding to the description segment and the inference segment is calculated. Based on the basic KL regularization parameter β, the dynamic weights are mapped to segment-specific KL regularization parameters β_s by combining the scaling factor α; Based on the segment-specific KL regularization parameters of the two segments, the preference margins of the description segment and the inference segment are weighted and combined to construct the segment-level joint loss function L_segment.

[0058] In one possible implementation, the third building module 603 is specifically used for: Extract the quality scores of preferred responses and rejection responses from domain expert-annotated data, and calculate the score difference Δ between them. ; By introducing a rating confidence parameter γ and using the rating gap as a correction term for the standard preference margin, a rating-aware direct preference optimization loss function L_rating is constructed, which adaptively adjusts the learning signal strength according to the size of the rating gap. For training samples without rating labels, the loss function is automatically degenerated into the standard direct preference optimization loss form.

[0059] In one possible implementation, the fourth building module 604 is specifically used for: The process of generating preference data is modeled as a causal structure that includes background factor U, and the background factor is identified as acting on the backdoor path formed by prompt construction, response generation and preference labeling. The backdoor adjustment method in causal inference is adopted. Marginalization calculation is performed based on the prior distribution p(U) of background factors to cut off the backdoor path and eliminate background confounding bias. During training, for each training batch, K background factor instances are sampled from the prior distribution of background factors, each instance is embedded into the conditional position of the prompt, and the log-likelihood probability of the policy model and the reference model is calculated respectively. The marginalized average of the calculation results of K sampled instances is used to construct the causal decontamination loss function L_causal.

[0060] In one possible implementation, the expression for the total joint loss function is: L_total = L_segment + λ1 L_rating+λ2 L_causal; Where L_segment is the segment-level joint loss function, L_rating is the rating-aware direct preference optimization loss function, L_causal is the causal deconfounding loss function, and λ1 and λ2 are module balancing coefficients, typically taking values ​​of 0.5 and 0.6 respectively, which are adjusted based on the convergence of the validation set. Training uses the AdamW optimizer with a learning rate set to 1×10⁻⁶. -6 Up to 5×10 -6 Each batch contains 32-128 pairs of preference samples, and the training rounds are 3-5 epochs. The training process is combined with a learning rate scheduling strategy of linear warm-up and cosine annealing.

[0061] In one possible implementation, the first building module 601 is specifically used for: Collect domain hint data covering multiple typical business scenarios, and label the background factors corresponding to each hint, such as task type, business attributes, hierarchical requirements, and security level requirements; A baseline large model, supervised and fine-tuned by a domain-specific professional corpus, is used to sample and generate multiple candidate responses for each prompt, and a preference pair candidate pool is constructed. Organizational experts conduct multi-dimensional quality assessments of candidate responses and provide sub-scores. Based on the comprehensive scores, pairwise preference ranking labels, overall score gaps, and segment-level score data are generated.

[0062] In one possible implementation, the trained alignment model is directly deployed in a specific domain decision support scenario. After receiving domain business queries in text form, it outputs a structured reasoning report containing scenario description, object identification, risk assessment, and action suggestions.

[0063] The aforementioned multi-granularity preference signal enhancement and alignment training device for large-scale models in specific domains achieves high-quality preference alignment for large-scale models within a direct preference optimization framework by systematically improving the granularity, density, and purity of preference signals through multi-dimensional improvements. By splitting long-chain inference responses into description and inference segments and implementing segment-level differentiated optimization, it can adaptively match the learning intensity of different semantic components, effectively avoiding the imbalance problem of overfitting simple content and under-optimization of core inference content, significantly improving the quality of long-chain logical inference. By fusing expert rating gap information to transform sparse binary ranking signals into continuous intensity signals, the model can precisely adjust the learning intensity based on response quality differences, reducing invalid parameter updates and effectively accelerating training convergence and improving training efficiency. Through a backdoor adjustment method based on causal inference to eliminate spurious correlation interference from background factors, it ensures that the preference strategies learned by the model truly reflect the quality level of the content itself, significantly improving the model's generalization ability and output fairness across different business scenarios. Simultaneously, the overall solution is fully compatible with existing large-scale model training infrastructure, requiring no modification to the model network architecture, resulting in low engineering implementation costs and convenient deployment.

[0064] like Figure 7 As shown, Figure 7 This is a schematic diagram of the composition structure of the electronic device 700 provided in the embodiments of this application. The electronic device 700 includes: The device 700 includes a processor 701, a storage medium 702, and a bus 703. The storage medium 702 stores machine-readable instructions executable by the processor 701. When the electronic device 700 is running, the processor 701 communicates with the storage medium 702 via the bus 703. The processor 701 executes the machine-readable instructions to perform the steps of the multi-granularity preference signal enhancement alignment training method for a large model in a specific domain as described in the embodiments of this application.

[0065] In practical applications, the various components in the electronic device 700 are coupled together via a bus 703. It is understood that the bus 703 is used to achieve communication between these components. In addition to a data bus, the bus 703 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 7 The general designated all buses as Bus 703.

[0066] The aforementioned electronic devices achieve high-quality preference alignment for large-scale professional models within the framework of direct preference optimization through multi-dimensional systematic improvements to preference signals, including granular refinement, density enhancement, and purity improvement. By splitting long-chain inference responses into description and inference segments and implementing segment-level differentiated optimization, the learning intensity of different semantic components can be adaptively matched, effectively avoiding the imbalance problem of overfitting simple content and under-optimization of core inference content, and significantly improving the quality of long-chain logical inference of the model. By fusing expert rating gap information to transform sparse binary ranking signals into continuous intensity signals, the model can accurately adjust the learning intensity according to the difference in response quality, reduce invalid parameter updates, and effectively accelerate training convergence speed and improve training efficiency. By eliminating spurious correlation interference caused by background factors through a backdoor adjustment method based on causal inference, the model ensures that the preference strategies learned by the model truly reflect the quality level of the content itself, greatly improving the model's generalization ability and output fairness across different business scenarios. At the same time, the overall solution is fully compatible with existing large-scale model training infrastructure, requires no modification to the model network architecture, and has low engineering implementation and transformation costs and is easy to implement.

[0067] This application also provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by at least one processor 701, the multi-granularity preference signal enhancement alignment training method for large models in a specific domain described in this application is implemented.

[0068] In some embodiments, the storage medium may be a magnetic random access memory (FRAM), a read-only memory (ROM), or a programmable read-only memory (PROM). Erasable Programmable Read-Only Memory (EPROM) Electrically Erasable Programmable Read-Only Memory (EEPROM) Read-only memory, flash memory, magnetic surface storage, optical disc, or CD-ROM ROM, Compact Disc Read It can be a memory such as a memory only; or it can be a device that includes one or any combination of the above-mentioned memories.

[0069] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0070] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0071] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0072] The aforementioned computer-readable storage medium achieves high-quality preference alignment for large-scale professional models within the framework of direct preference optimization through multi-dimensional systematic improvements to the preference signal, including granular refinement, density enhancement, and purity enhancement. By splitting long-chain inference responses into description and inference segments and implementing segment-level differentiated optimization, it can adaptively match the learning intensity of different semantic components, effectively avoiding the imbalance problem of overfitting simple content and under-optimization of core inference content, and significantly improving the quality of long-chain logical inference of the model. By fusing expert rating gap information to transform sparse binary ranking signals into continuous intensity signals, the model can accurately adjust the learning intensity according to the difference in response quality, reduce invalid parameter updates, and effectively accelerate training convergence speed and improve training efficiency. By eliminating spurious correlation interference caused by background factors through a backdoor adjustment method based on causal inference, it ensures that the preference strategies learned by the model truly reflect the quality level of the content itself, greatly improving the model's generalization ability and output fairness across different business scenarios. At the same time, the overall solution is fully compatible with existing large-scale model training infrastructure, requires no modification to the model network architecture, and has low engineering implementation costs and is easy to implement.

[0073] In the several embodiments provided in this application, it should be understood that the disclosed methods and electronic devices can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0074] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0075] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0076] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0077] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-granularity preference signal enhancement and alignment training method for large-scale models in a specific domain, characterized in that, Includes the following steps: Construct a domain-specific preference training dataset, in which each sample contains domain task prompts, pairs of preferred and rejected responses, expert quality scores, and annotations of task context factors; Semantic segmentation is performed on each pair of candidate responses, and the long-chain inference response of a specific domain is split into description segment and inference segment. Dynamic optimization weights are calculated based on segment-level score differences, segment-specific KL regularization parameters are mapped and generated, and segment-level joint loss is constructed. The expert rating difference between paired responses is extracted, and the rating gap is incorporated into the optimization objective as a marginal correction term for preference. A direct preference optimization loss based on rating perception is constructed, and the binary ranking signal is transformed into a continuous intensity signal. Based on the causal inference backdoor adjustment method, the background factors that affect preference labeling are marginalized, the spurious correlation backdoor path introduced by the background factors is cut off, and a causal deconfounding loss is constructed. The total joint loss is obtained by weighting and summing the segment-level joint loss, rating perception loss, and causal decontamination loss according to a preset balance coefficient. Based on the total joint loss, the parameters of the large model in a specific domain are iteratively optimized to complete the preference alignment training.

2. The multi-granularity preference signal enhancement and alignment training method for large-scale models in a specific domain, as described in claim 1, is characterized in that... The process of semantic segmentation for each pair of candidate responses, splitting long-chain inference responses in a specific domain into description segments and inference segments, calculating dynamically optimized weights based on segment-level score differences, mapping and generating segment-specific KL regularization parameters, and constructing segment-level joint loss includes: By using predefined structured tags or rule segmenters, each long chain reasoning response is automatically divided into a descriptive segment that presents objective facts about scene elements, domain object attributes and parameters, and a reasoning segment that is used for execution scheme deduction and multi-hop logic analysis. Based on a domain-specific comprehensive scoring function, the quality scores of the description segment and the reasoning segment in the preferred response and the rejection response are calculated respectively, and the preference difference ΔR_desc of the description segment and the preference difference ΔR_reason of the reasoning segment are obtained. Introducing the temperature parameter τ, the preference difference between the two segments is normalized using the softmax function, and the dynamic optimization weight w_s corresponding to the description segment and the inference segment is calculated. Based on the basic KL regularization parameter β, the dynamic weights are mapped to segment-specific KL regularization parameters β_s by combining the scaling factor α; Based on the segment-specific KL regularization parameters of the two segments, the preference margins of the description segment and the inference segment are weighted and combined to construct the segment-level joint loss function L_segment.

3. The multi-granularity preference signal enhancement and alignment training method for large-scale models in a specific domain, as described in claim 2, is characterized in that... The step of extracting the expert rating difference between paired responses, incorporating the rating gap as a preference marginal correction term into the optimization objective, constructing a rating-aware direct preference optimization loss, and transforming the binary ranking signal into a continuous intensity signal includes: Extract the quality scores of preferred responses and rejection responses from domain expert-annotated data, and calculate the score difference Δ between them. ; By introducing a rating confidence parameter γ and using the rating gap as a correction term for the standard preference margin, a rating-aware direct preference optimization loss function L_rating is constructed, which adaptively adjusts the learning signal strength according to the size of the rating gap. For training samples without rating labels, the loss function is automatically degenerated into the standard direct preference optimization loss form.

4. The multi-granularity preference signal enhancement and alignment training method for large-scale models in a specific domain, as described in claim 1, is characterized in that... The causal inference-based backdoor adjustment method marginalizes background factors influencing preference labeling, cuts off spurious backdoor paths introduced by background factors, and constructs a causal decontamination loss, including: The process of generating preference data is modeled as a causal structure that includes background factor U, and the background factor is identified as acting on the backdoor path formed by prompt construction, response generation and preference labeling. The backdoor adjustment method in causal inference is adopted. Marginalization calculation is performed based on the prior distribution p(U) of background factors to cut off the backdoor path and eliminate background confounding bias. During training, for each training batch, K background factor instances are sampled from the prior distribution of background factors, each instance is embedded into the conditional position of the prompt, and the log-likelihood probability of the policy model and the reference model is calculated respectively. The marginalized average of the calculation results of K sampled instances is used to construct the causal decontamination loss function L_causal.

5. The multi-granularity preference signal enhancement and alignment training method for large-scale models in a specific domain, as described in claim 1, is characterized in that... The expression for the total joint loss function is: L_total=L_segment+λ1 L_rating+λ2 L_causal; Where L_segment is the segment-level joint loss function, L_rating is the rating-aware direct preference optimization loss function, L_causal is the causal deconfounding loss function, and λ1 and λ2 are module balancing coefficients, typically taking values ​​of 0.5 and 0.6 respectively, which are adjusted based on the convergence of the validation set. Training uses the AdamW optimizer with a learning rate set to 1×10⁻⁶. -6 Up to 5×10 -6 Each batch contains 32-128 pairs of preference samples, and the training rounds are 3-5 epochs. The training process is combined with a learning rate scheduling strategy of linear warm-up and cosine annealing.

6. The multi-granularity preference signal enhancement and alignment training method for large-scale models in a specific domain, as described in claim 1, is characterized in that... The construction of the domain-specific preference training dataset includes: Collect domain hint data covering multiple typical business scenarios, and label the background factors corresponding to each hint, such as task type, business attributes, hierarchical requirements, and security level requirements; A baseline large model, supervised and fine-tuned by a domain-specific professional corpus, is used to sample and generate multiple candidate responses for each prompt, and a preference pair candidate pool is constructed. Organizational experts conduct multi-dimensional quality assessments of candidate responses and provide sub-scores. Based on the comprehensive scores, pairwise preference ranking labels, overall score gaps, and segment-level score data are generated.

7. The multi-granularity preference signal enhancement and alignment training method for large-scale models in a specific domain, as described in claim 1, is characterized in that... Once trained, the alignment model is directly deployed in a specific domain decision support scenario. After receiving domain business queries in text form, it outputs a structured reasoning report containing scenario description, object identification, risk assessment, and action suggestions.

8. A multi-granularity preference signal enhancement and alignment training device for large models in a specific domain, characterized in that, The device includes: The first building module is used to build a domain-specific preference training dataset. Each sample contains domain task prompts, pairs of preferred and rejected responses, expert quality score data, and task context factor annotations. The second construction module is used to perform semantic segmentation on each pair of candidate responses, split the long chain inference response of a specific domain into description segments and inference segments, calculate dynamic optimization weights based on segment-level score differences, map and generate segment-specific KL regularization parameters, and construct segment-level joint loss. The third construction module is used to extract the expert rating difference between paired responses, incorporate the rating gap as a preference marginal correction term into the optimization objective, construct the rating-perceived direct preference optimization loss, and transform the binary ranking signal into a continuous intensity signal. The fourth module is used to marginalize background factors that influence preference labeling based on the causal inference backdoor adjustment method, cut off the spurious correlation backdoor path introduced by background factors, and construct causal deconfounding loss. The iterative optimization module is used to sum the segment-level joint loss, rating-aware loss, and causal decontamination loss by weighting them with preset balance coefficients to obtain the total joint loss. Based on the total joint loss, the parameters of the large model in a specific domain are iteratively optimized to complete the preference alignment training.

9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the multi-granularity preference signal enhancement alignment training method for a domain-specific large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the multi-granularity preference signal enhancement alignment training method for a domain-specific large model as described in any one of claims 1 to 7.

Images