Large Model Alignment System Based on Multi-Level Cascade Dynamic Supervision

Abstract

This invention discloses a large model alignment system based on multi-layered cascaded dynamic supervision, relating to the fields of artificial intelligence security and alignment technology. Through refined hierarchical division of labor, the invention rapidly establishes security boundaries in the first layer while progressively deepening alignment in the second and third layers, avoiding performance degradation caused by over-supervision. The multi-layered cascaded architecture provides a complete supervision trajectory from behavior to value. The self-other overlapping representation in the second layer and the inference chain generation in the third layer provide interpretable intermediate states for the alignment process, significantly improving the system's transparency. The meta-learning framework and dynamic imbalance evaluation mechanism in the third layer enable the system to effectively address out-of-distribution ethical challenges. A strong-weak capability feedback mechanism achieves the co-evolution of the supervisor's capabilities and the supervised model's capabilities. This system adopts a modular design, supports progressive deployment, and allows for flexible configuration of the three layers of supervision strength according to computing resources and application scenarios.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security and alignment technology, and in particular to a large model alignment system based on multi-layered cascaded dynamic supervision. Background Technology

[0002] With the rapid improvement of the capabilities of large-scale language models, artificial intelligence (AI) alignment faces increasingly severe challenges. Traditional human feedback reinforcement learning (RLHF) methods rely on human annotators to provide supervision signals, but they suffer from two fundamental drawbacks.

[0003] First, there is the bottleneck of supervision capabilities. Existing research has proposed methods using weak models (such as GPT-2) to supervise strong models (such as GPT-4), but this only improves performance by less than 10% in ethical tasks (safety, fairness, transparency, and accountability, etc.) involved in the development, deployment, and use of artificial intelligence, revealing a lack of generalization. This deficiency stems from the limitations of weak models in handling complex ethical scenarios, failing to fully capture the intrinsic motivations of strong models. As AI capabilities increase dramatically, traditional supervision methods will become ineffective, but how weak supervisors can capture the subtle deceptive behavior of strong models remains unresolved.

[0004] Secondly, there is the alignment paradox. Insufficient supervision leads to model runaway, while excessive supervision impairs the model's core intelligent capabilities, including cognitive performance such as reasoning, understanding, and creation, as well as task execution. This creates a dilemma: how to maintain model performance while ensuring alignment.

[0005] According to tests conducted by relevant research institutions, up to 85% of advanced commercial models attempt to disable their monitoring mechanisms and subsequently deny such behavior in specific scenarios.

[0006] The common problem with existing technologies is that they use a single-level supervisory signal, lack a progressive constraint mechanism from behavior and cognition to value, and the supervisory capabilities are fixed and cannot grow in sync with the evolution of AI. Therefore, there is an urgent need for a new system that can establish a multi-layered cascaded supervisory chain and dynamically improve the supervisor's capabilities through a capability feedback mechanism. Summary of the Invention

[0007] The purpose of this invention is to provide a large model alignment system based on multi-layered cascaded dynamic supervision (MCS). Through a multi-layered progressive supervision chain and a strong-weak capability feedback mechanism, it breaks through the dilemma of weak supervision and achieves synchronous growth of supervision capability and AI evolution.

[0008] The technical solution adopted in this invention is as follows:

[0009] A large model alignment system based on multi-layered cascaded dynamic supervision includes: an input module, a behavior supervision layer (L1), a cognitive supervision layer (L2), a value supervision layer (L3), and a cascaded fusion module;

[0010] The input module is used to input the base model, the safety behavior dataset, the intent reasoning dataset, and the value alignment dataset; among which, the base model is a large pre-trained language model;

[0011] The behavior supervision layer uses rule matching and statistical features to set the behavior supervision loss function, performs supervised fine-tuning of the base model based on the safety behavior dataset, and performs real-time filtering and correction of the content generated by the model, so that the model learns basic defensive behavior patterns.

[0012] The cognitive supervision layer integrates a self-other overlap (SOO) representation module. Based on the self-other intention alignment mechanism and intention reasoning regularization term, the cognitive supervision loss function is set. The base model after supervision fine-tuning is trained on the intention reasoning dataset. By analyzing the self-other overlap representation, the hidden deceptive behavior in the user input content is identified. Based on the reverse reasoning chain, the model is made to explicitly generate the user's implicit intention, thereby dynamically correcting the semantic value of the content generated by the model, strengthening the model's ability to understand complex social situations, and reducing alignment failure caused by misunderstanding from the root.

[0013] The value supervision layer adopts a value supervision loss function based on the dynamic imbalance benchmark (DMB) evaluation model and the policy entropy regularization term. Based on the value alignment dataset, the base model trained on intention reasoning is trained with the meta-learning (MAML) framework for cross-scenario generalization training to train the model's ability to judge the value of the target scenario and ensure that the model can maintain the robustness of value alignment when facing out-of-distribution ethical challenges.

[0014] The cascaded fusion module employs a three-stage dynamic adjustment strategy to perform end-to-end optimization of the base model after cross-scenario generalization training. This includes: calculating the current supervised loss of the model based on the safety behavior dataset and supervised loss function at the behavior supervision layer; calculating the current cognitive supervised loss of the model based on the intention reasoning dataset and cognitive supervised loss function at the cognitive supervision layer; and calculating the current value supervised loss of the model based on the value alignment dataset and value supervised loss function at the value supervision layer. Based on the loss weights configured for each stage, the behavior supervision loss, cognitive supervision loss, and value supervision loss of the current stage are weighted and fused to obtain the total supervised loss. End-to-end optimization of the base model for the current stage is then performed based on the total supervised loss. Among these, the behavior supervision loss in the first stage has the highest loss weight; the cognitive supervision loss in the second stage has the highest loss weight; and the value supervision loss in the third stage has the highest loss weight.

[0015] Furthermore, basic defensive behavioral patterns include rejection, rewriting, and guidance.

[0016] Furthermore, supervised fine-tuning adopts a low-rank adaptive (LoRA) approach, intent reasoning training adopts a distributed training framework, and cross-scenario generalization training adopts a multi-round iterative optimization and red team testing verification approach.

[0017] Furthermore, the behavior supervision loss function is:

[0018]

[0019] in, These are the model parameters for the base model. For mathematical expectation, Input for the model (i.e., user input). Output for the model (i.e., the content generated by the model). For a safety behavior dataset, This represents the policy distribution of the base model, i.e., the probability distribution of the model output y given a model input x. Wherein, , This is the output weight matrix of the base model. This represents the final hidden layer feature of the base model. for function, Let KL (Kullback–Leibler) divergence function be used. The first regularization coefficient is... The policy distribution of the reference model can be set as the original base model or the base model before the previous round of supervised fine-tuning; The expression can be set as: . The output weight matrix of the reference model. This refers to the final hidden layer features of the reference model.

[0020] Furthermore, the behavioral supervision function also includes KL divergence constraints, i.e. The value is less than a preset threshold to prevent the adaptation process from deviating from the original capability distribution and to ensure that the policy distribution after task adaptation does not significantly deviate from the initial distribution obtained by meta-learning.

[0021] Furthermore, the cognitive supervision loss function is:

[0022]

[0023] in, Input to the model, For model output, For contextual information, For intent reasoning datasets, This represents the intermediate hidden layer of the model. The output of the self-other overlap representation module, i.e., the model self-representation vector. Compared with the inferred representation vector of others The degree of overlap can be based on vectors. and The cosine similarity between them represents their degree of overlap; This is an intent reasoning regularization term, used to penalize the model for generating outputs that are inconsistent with the inferred intent. This is the second regularization coefficient. Wherein, ,or ,in, This is a preset normalization factor. For the intent reasoning regularization term, it can be set to: ,in, The true intent label is used for annotation. These are the parameters for the intent classifier. Alternatively, a contrastive loss form can be used: To ensure that the output has a higher similarity to the true intent than negative samples, For preset safety boundary values, For similarity function, The intent vector for positive samples. The intention vector for negative samples.

[0024] Furthermore, the value supervision loss function is:

[0025]

[0026] in, For the first The evaluation scores for each evaluation dimension (such as fairness score, honesty score, etc.). The total number of evaluation dimensions. For the first The weights of each evaluation dimension, and , ; For policy entropy regularization, This is the third regularization coefficient; where the evaluation score can be calculated as follows: ,in, Value-aligned datasets The Middle A dataset with several evaluation dimensions. For the first Evaluation functions for each evaluation dimension (such as accuracy, F1 score (harmonic mean of accuracy and recall) etc.). Indicates the corresponding model input The tag. The expression for the policy entropy regularization term can be set as: This encourages output diversity and prevents excessive model determinism from leading to fragility.

[0027] Furthermore, in the three-stage dynamic control strategy, the loss weights of behavioral supervision loss, cognitive supervision loss, and value supervision loss in the first stage are set to 0.5, 0.3, and 0.2, respectively; the loss weights of behavioral supervision loss, cognitive supervision loss, and value supervision loss in the second stage are set to 0.3, 0.4, and 0.3, respectively; and the loss weights of behavioral supervision loss, cognitive supervision loss, and value supervision loss in the third stage are set to 0.2, 0.3, and 0.5, respectively.

[0028] Furthermore, it includes a strong-weak capability feedback module. Through knowledge distillation and transfer learning, high-level capabilities learned in the value supervision layer (such as reasoning ability for target tasks, abstract knowledge, and value judgment patterns) are transferred to the behavioral and cognitive supervision layers. This dynamically expands the capability boundaries of the supervisors used for fine-tuning (weak supervisors) and intention reasoning training (weak supervisors). This allows the supervisor's capabilities to grow synchronously with the evolution of the supervised model, breaking through the bottleneck of fixed supervisory capabilities in traditional methods.

[0029] Furthermore, the transfer of advanced skills learned in the value supervision layer to the behavioral and cognitive supervision layers specifically includes:

[0030] The model output distribution in the target task is used as a soft label. The supervisors for supervised fine-tuning and intention reasoning training are trained based on knowledge distillation. During distillation, the teacher / student distribution is softened by temperature T and compensated by T² weighted gradient. The temperature T is 2-5, with 3 being the preferred value.

[0031] The last hidden layer features of the model in the target task (i.e., high-dimensional feature representation) is mapped to the representation spaces of the cognitive supervision layer and the behavioral supervision layer respectively through linear mapping or neural networks. ;

[0032] The supervisor is trained using behavior supervision loss, distillation loss, and feature transfer loss for supervision fine-tuning.

[0033] The training is based on cognitive supervision loss, distillation loss, and feature transfer loss, along with a supervisor for intention reasoning training.

[0034] Furthermore, when training the supervisor for supervised fine-tuning / intent reasoning training, the weights of distillation loss and feature transfer loss are dynamically adjusted:

[0035] The supervisory capabilities (e.g., accuracy) of the behavioral and cognitive supervisory layers are periodically evaluated on a standard alignment test set. The weights of the distillation loss and feature transfer loss are dynamically adjusted based on the evaluation results. Furthermore, the execution frequency N of the strong-weak capability feedback module (i.e., once every N training steps) can be dynamically adjusted based on the evaluation results. For example, ,in, , These are the weights before and after adjustment, respectively. The adjustment coefficient is used to adjust the weights to a reasonable range of values. For the preset target performance threshold, This is the combined evaluation result of the two supervisors.

[0036] Furthermore, the distillation loss can be specifically set as follows:

[0037] in, This represents the output distribution of the teacher model (i.e., the base model used by the value supervision layer). This represents the output distribution of the student model (i.e., the supervisor of the supervised fine-tuning / intent reasoning training).

[0038] Furthermore, the feature transfer loss is set to That is, minimizing the representation distance during training. Wherein, This represents a linear mapping matrix from the value supervision layer to the cognitive supervision layer. This represents a linear mapping matrix from the value supervision layer to the behavior supervision layer. Preferably, , and The matrix shapes are 4096, 2048, and 512, respectively.

[0039] The technical solution provided by this invention brings at least the following beneficial effects:

[0040] (1) Balancing alignment and performance, the three-layer cascaded supervision, through a refined hierarchical division of labor, quickly establishes a safety boundary at the L1 layer while the L2 and L3 layers gradually deepen the alignment, avoiding performance degradation caused by over-supervision;

[0041] (2) Improve alignment robustness. The meta-learning framework of the L3 layer and the dynamic evaluation mechanism of DMB enable the system to effectively cope with off-distribution ethical challenges.

[0042] (3) Breaking through the bottleneck of supervision capability, this invention achieves the co-evolution of the supervisor capability and the supervised model capability through a strong-weak capability feedback mechanism, fundamentally solving the problem that weak supervisors are difficult to constrain strong models;

[0043] (4) Enhanced interpretability: The multi-layered cascaded architecture provides a complete supervision trajectory from behavior to value. The SOO representation of the L2 layer and the inference chain generation of the L3 layer provide interpretable intermediate states for the alignment process, which significantly improves the transparency of the system.

[0044] (5) The project is highly feasible. The system adopts a modular design, supports progressive deployment, and can flexibly configure the strength of the three-layer supervision according to computing resources and application scenarios. It supports efficient parameter fine-tuning techniques such as LoRA, and can train 72B+ parameter models on medium-sized GPU (graphics processing unit) clusters. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described in detail and completely. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments.

[0046] This invention provides a large model alignment system based on multi-layer cascaded dynamic supervision, including: an input module, a behavior supervision layer (L1), a cognitive supervision layer (L2), a value supervision layer (L3), a cascaded fusion module, and a strong-weak capability feedback module. It constructs a three-layer progressive supervision system through layers L1 to L3. Each layer has an independent supervision target and achieves inter-layer collaborative optimization through a cascade mechanism.

[0047] The input module is used to input the base model (a pre-trained large-scale language model), the safe behavior dataset, the intent reasoning dataset, and the value alignment dataset. The safe behavior dataset consists of safe-dangerous behavior comparison data, covering multiple risk scenarios; the intent reasoning dataset contains multi-turn dialogue samples and standardizes user implicit intent and self-other overlap (SOO) representation of target values; the value alignment dataset includes multiple complex ethical dilemma scenarios and standard multi-dimensional moral scores (such as fairness, honesty, harmlessness, etc.).

[0048] Layer L1 focuses on the surface compliance of the model output. By constructing a fine-grained set of behavioral constraint rules, it ensures that the model's response conforms to basic safety principles. This layer uses a combination of rule matching and statistical features to filter and correct the model-generated content in real time based on a safe behavior dataset. The behavioral supervision loss function of Layer L1, i.e., the behavioral deviation function, is calculated based on the divergence measure between the model output (the model's generated content) and the distribution of expected human behavior. This layer uses large-scale safe-dangerous behavior comparison data for training and the behavioral supervision loss function to supervise and fine-tune the base model, filtering and correcting the model-generated content in real time, enabling the model to quickly learn basic defensive behavioral patterns such as refusal to answer, rewriting, and guidance.

[0049] Layer L2 delves deeper into the model's internal representation space, focusing on optimizing the model's ability to understand and reason about user intent. This layer uses the SOO representation as the core supervision signal, quantifying the model's self-state (corresponding to the self-representation vector). Others' mental states (corresponding to inferred other representation vectors) The L2 layer enhances the model's ability to compute empathy by increasing its discriminative power. The L2 layer is based on a cognitive supervision loss function. The intention reasoning dataset is used to train the supervised fine-tuned base model for intention reasoning. By analyzing the self-other overlap representation, hidden deceptive behaviors in user input are identified. Based on the inverse reasoning chain, the model is made to explicitly generate the user's implicit intention, thereby dynamically correcting the semantic value of the model-generated content, strengthening the model's ability to understand complex social situations, and reducing alignment failures caused by misunderstandings from the root.

[0050] Layer L3 systematically evaluates the model's performance in complex ethical dilemmas using a Dynamic Imbalance Benchmark (DMB). This layer not only examines the model's explicit output but also probes its internal activation patterns through interpretability techniques to identify potential risks of deceptive alignment. Based on a value alignment dataset, this layer employs a meta-learning (MAML) framework for cross-scenario generalization training of the base model trained on intention reasoning. This trains the model's value judgment ability for target scenarios, ensuring the model maintains robustness in value alignment when facing out-of-distribution ethical challenges.

[0051] In one embodiment, the three-layer progressive supervision system built based on this method adopts a three-stage progressive training strategy, specifically as follows:

[0052] Basic alignment phase: L1 layer supervised fine-tuning is performed on the base model, and efficient methods such as LoRA (Low-Rank Adaptation) are used to reduce computational overhead. The basic safety boundary is established after 10-15 training rounds.

[0053] Cognitive reinforcement stage: L2 layer cognitive supervision is introduced, combined with SOO representation learning and intent reasoning chain generation, and a distributed training framework is used to complete the training on a GPU cluster for 15-20 rounds to reinforce intent understanding.

[0054] Value deepening stage: Integrate L3 layer value supervision, conduct multiple rounds of iterative optimization and red team testing and verification through DMB benchmark, and train for 10-15 rounds to achieve deep value alignment.

[0055] The cascaded fusion module employs a three-stage dynamic adjustment strategy to perform end-to-end optimization of the base model after cross-scenario generalization training. This includes: calculating the model's current supervised loss at the behavior supervision layer based on a safety behavior dataset and a supervised loss function; calculating the model's current cognitive supervised loss at the cognitive supervision layer based on an intent reasoning dataset and a cognitive supervised loss function; and calculating the model's current value supervised loss at the value supervision layer based on a value alignment dataset and a value supervised loss function. Based on the loss weights configured for each stage, the behavior supervision loss, cognitive supervision loss, and value supervision loss for the current stage are weighted and fused to obtain the total supervised loss. The base model is then optimized end-to-end for the current stage based on this total supervised loss. That is, the total supervised loss function is: Loss weights of the three losses A dynamic adjustment strategy is adopted, adaptively adjusting the weights based on the training phase and model performance, with the sum of the three loss weights being 1. Specifically, in the first phase (initial phase), the behavioral supervision loss of the L1 layer is given a higher weight, for example... To quickly establish a basic security boundary; in the second phase (mid-term phase), the loss weight of the cognitive supervision loss in the L2 layer is gradually increased, for example... To enhance intent understanding capabilities; the third stage (late stage) focuses on the loss weight of value supervision loss at layer L3, for example... This cascaded architecture achieves deep value alignment by enabling information flow between layers through gradient backpropagation. This allows shallow supervision to provide a stable foundation for deeper layers, while deep feedback guides the continuous refinement of shallow layers, forming a positive iterative cycle.

[0056] The strong-weak capability feedback module aims to transfer and "feed back" the advanced capabilities, abstract concepts, and complex reasoning chains learned by the supervised strong model to the weaker supervisor, thereby dynamically increasing the supervisor's capability ceiling and fundamentally overcoming the alignment paradox. This module, through knowledge distillation and transfer learning, transfers advanced capabilities learned in the value supervision layer (such as reasoning ability for target tasks, abstract knowledge, and value judgment patterns) to the behavioral and cognitive supervision layers, dynamically expanding the capability boundaries of the supervisors undergoing fine-tuning (weak supervisors) and those trained on intentional reasoning (weak supervisors). This allows the supervisor's capability to grow synchronously with the evolution of the supervised model, breaking the bottleneck of fixed supervisory capabilities in traditional methods. In this embodiment, the specific implementation steps include:

[0057] (1) Acquisition of advanced capabilities: At the L3 level, high capability models (such as GPT-4 level or higher, referred to as L3 model, i.e. teacher model) are used to handle complex ethical dilemma tasks, and record the complete reasoning trajectory, intermediate decision nodes and final value judgments.

[0058] (2) Knowledge distillation: the output distribution of the L3 model As a soft label, it is based on the supervisory model trained with L2 and L1 layers. The corresponding distillation loss function is ;

[0059] (3) Feature transfer: The high-dimensional features learned by the L3 layer are represented by the transfer learning method. Mapping to the representation spaces of L2 and L1 layers via linear mapping or neural networks. To minimize the representation distance, the corresponding feature transfer loss is: ;

[0060] (4) Capability assessment and iteration: Periodically assess the improvement of the supervisory capabilities of the L2 and L1 layers on the standard alignment test set, and dynamically adjust the feedback intensity (i.e., distillation weight) and execution frequency N (one feedback is performed every N training steps) based on the assessment results.

[0061] The aforementioned strong-weak capability feedback mechanism enables the supervisor's capabilities to grow synchronously with the evolution of the supervised model, breaking the bottleneck of fixed supervisory capabilities in traditional methods. In practical implementation, it can also be flexibly configured based on the actual scenario; both L1 and L2 layers can include knowledge distillation and feature transfer, or only one of them can be executed.

[0062] Example: System Deployment Based on the Large Language Model Llama 3.1-70B

[0063] (1-1) Model and Environment Preparation

[0064] The Llama 3.1-70B was chosen as the base model, and INT4 (4-bit wide integer) quantization technology was used to compress the video memory usage from 145GB to 36GB. A distributed training environment was built using 6 GPUs (total video memory 288GB).

[0065] (1-2) Dataset Construction

[0066] Construct three types of datasets to support three layers of supervision respectively:

[0067] L1 Safety Behavior Dataset: Contains 50,000 pairs of safe-dangerous behavior samples, covering 12 risk scenarios;

[0068] L2 Intent Reasoning Dataset: Contains 30,000 multi-turn dialogue samples, labeled with user implicit intent and SOO representation target value;

[0069] L3 Value Alignment Dataset: Contains 20,000 complex ethical dilemma scenarios, labeled with multi-dimensional moral scores (fairness, honesty, harmlessness, etc.).

[0070] (1-3) Implementation of the three-stage training

[0071] Basic alignment phase (10 rounds): Only L1 layer is activated, and the learning rate is set to... Batch size 32, using LoRA rank Make fine-tuning to establish basic security boundaries;

[0072] Cognitive reinforcement phase (15 rounds): Introducing the L2 layer and configuring weights. The learning rate decayed to The integration of the SOO representation module enhances cognitive supervision. In specific processing, this process does not involve the L3 layer. If the total supervision loss function is used... Then, the value supervision loss of the corresponding L3 layer can be directly set to 0. Furthermore, at this stage, the fine-tuned base model can also be directly trained for intent reasoning based on the cognitive supervision loss function.

[0073] Value Deepening Stage (10 rounds): Activate the complete three-layer architecture. At this stage, when performing cross-scenario generalization training, the training loss function can be either the cognitive supervision loss function or the total supervision loss function. The corresponding weight configuration is as follows: The learning rate dropped to Value alignment is optimized through DMB benchmark iteration.

[0074] (1-4) Deployment of Capability Feedback Mechanism

[0075] Starting from the value deepening stage, a feedback operation is performed once every 500 training steps, i.e., the execution frequency. The initial value is 500:

[0076] 1) Process 100 complex ethical scenarios using the full 70B model at the L3 layer, and record the output distribution and feature representation;

[0077] 2) Train the L2 layer supervisor using knowledge distillation (approximately 7 bytes of parameters), and distill the loss weights. The initial value is 0.3;

[0078] 3) Use a feature transfer method to map the L3 representation to the L1 layer (approximately 700M parameters), with transfer loss weights. The initial value is 0.2;

[0079] 4) Evaluate the supervision capabilities of L2 and L1 on the standard test set to dynamically adjust... and ,as well as .

[0080] (1-5) Effect Evaluation

[0081] A comprehensive evaluation was conducted on the DMB dynamic imbalance benchmark, and the results showed that the spoofing detection rate ( Compared to the RLHF baseline, the performance decreased by 58%, and the capability retention was ( The alignment fidelity reached 87%, and the overall performance reached 78% in highly competitive tasks, demonstrating comprehensive effectiveness. .in, , The number of correctly detected fraudulent samples. This represents the number of deceptive samples that were missed during detection. ,in, To align the accuracy on the benchmark task after training, This represents the accuracy of the original base model.

[0082] Supervisor capability improvement assessment: The accuracy of the L2 layer on the complex intent reasoning task improved from 65% to 82%, and the F1 score of the L1 layer on the covert deception detection task improved from 0.71 to 0.85.

[0083] The system proposed in this invention can be widely applied in various fields requiring highly reliable AI systems, such as autonomous driving, medical diagnosis, and intelligent customer service. After adopting the system proposed in this invention, the accuracy of the supervisor's judgment in complex ethical tasks is improved by more than 25%; the performance retention rate of each layer of supervision reaches more than 85%, significantly better than the approximately 70% of traditional single-layer supervision methods; in strong adversarial task testing, the alignment fidelity of this invention reaches more than 75%, an improvement of 20 percentage points compared to the baseline method. Based on the proposed large-model alignment system with multi-layer cascaded dynamic supervision, this invention overcomes the dilemma of weak supervision through a three-layer progressive supervision chain and a strong-weak capability feedback mechanism, achieving synchronous growth of supervision capability and AI evolution.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0085] The above descriptions are merely some embodiments of the present invention. For those skilled in the art, various modifications and improvements can be made without departing from the inventive concept of the present invention, and all such modifications and improvements fall within the scope of protection of the present invention.

Claims

1. A large model alignment system based on multi-layered cascaded dynamic supervision, characterized in that, Includes the following steps: A large model alignment system based on multi-layered cascaded dynamic supervision includes: an input module, a behavior supervision layer, a cognitive supervision layer, a value supervision layer, and a cascaded fusion module; The input module is used to input the base model, the safety behavior dataset, the intent reasoning dataset, and the value alignment dataset; among which, the base model is a large pre-trained language model; The behavior supervision layer uses rule matching and statistical features to set the behavior supervision loss function, performs supervised fine-tuning of the base model based on the safety behavior dataset, and filters and corrects the content generated by the model in real time, so that the model learns basic defensive behavior patterns. The cognitive supervision layer integrates a self-other overlapping representation module, sets a cognitive supervision loss function based on the self-other intention alignment mechanism and intention reasoning regularization term, trains the base model after supervision fine-tuning based on the intention reasoning dataset, identifies hidden deceptive behaviors in user input by analyzing the self-other overlapping representation degree, and allows the model to explicitly generate the user's implicit intention based on the inverse reasoning chain. The value supervision layer adopts the value supervision loss function based on the dynamic imbalance benchmark evaluation model DMB and the policy entropy regularization term. Based on the value alignment dataset, the base model trained on intention reasoning is trained with a meta-learning framework to perform cross-scene generalization training in order to train the model's ability to judge the value of the target scene. The cascaded fusion module employs a three-stage dynamic adjustment strategy to perform end-to-end optimization of the base model after cross-scenario generalization training. This includes: calculating the current supervised loss of the model at the behavioral supervision layer, calculating the current cognitive supervision loss of the model at the cognitive supervision layer, and calculating the current value supervision loss of the model at the value supervision layer; based on the loss weights configured for each stage, the behavioral supervision loss, cognitive supervision loss, and value supervision loss of the current stage are weighted and fused to obtain the total supervised loss; and the base model is then optimized end-to-end for the current stage based on the total supervised loss. Among these, the behavioral supervision loss in the first stage has the highest loss weight; the cognitive supervision loss in the second stage has the highest loss weight; and the value supervision loss in the third stage has the highest loss weight.

2. The large model alignment system based on multi-layer cascaded dynamic supervision as described in claim 1, characterized in that, Basic defensive behavioral patterns include rejection, rewriting, and guidance.

3. The large model alignment system based on multi-layer cascaded dynamic supervision as described in claim 1, characterized in that, Supervised fine-tuning adopts a low-rank adaptive approach, intent reasoning training adopts a distributed training framework, and cross-scenario generalization training adopts a multi-round iterative optimization and red team testing and verification approach.

4. The large model alignment system based on multi-layer cascaded dynamic supervision as described in claim 1, characterized in that, The behavior supervision loss function is: ； in, These are the model parameters for the base model. For mathematical expectation, Enter content for the user. Generate content for the model. For a safety behavior dataset, For the policy distribution of the base model, Let KL divergence function be used. The first regularization coefficient is... The policy distribution is the reference model, which is either the original base model or the base model before the previous round of supervised fine-tuning.

5. The large model alignment system based on multi-layer cascaded dynamic supervision as described in claim 4, characterized in that, The behavior supervision loss function also includes the KL divergence constraint, i.e. Less than the preset threshold.

6. The large model alignment system based on multi-layer cascaded dynamic supervision as described in claim 1, characterized in that, The cognitive supervision loss function is: ； in, These are the model parameters for the base model. Enter content for the user. Generate content for the model. For contextual information, For intent reasoning datasets, This represents the intermediate hidden layer of the model. The output of the self-other overlap representation module, i.e., the model self-representation vector. Compared with the inferred representation vector of others The degree of overlap, For intention reasoning regularization, This is the second regularization coefficient.

7. The large model alignment system based on multi-layer cascaded dynamic supervision as described in claim 1, characterized in that, The value-monitored loss function is: ； in, These are the model parameters for the base model. For the first The evaluation scores for each evaluation dimension, The total number of evaluation dimensions. For the first The weights of each evaluation dimension, and , ; For policy entropy regularization, This is the third regularization coefficient.

8. The large model alignment system based on multi-layer cascaded dynamic supervision as described in claim 1, characterized in that, It also includes a strong-weak capability feedback module, which, through knowledge distillation and transfer learning, transfers advanced capabilities learned in the value supervision layer to the behavioral supervision layer and the cognitive supervision layer, so as to dynamically expand the capability boundaries of the supervisors for supervision fine-tuning and the supervisors for intention reasoning training.

9. The large model alignment system based on multi-layer cascaded dynamic supervision as described in claim 8, characterized in that, The transfer of advanced skills learned in the value supervision layer to the behavioral and cognitive supervision layers specifically includes: The model output distribution in the target task is used as a soft label. The supervisors trained by knowledge distillation and intention reasoning are trained separately. During distillation, the teacher / student distribution is softened by temperature T and compensated by T² weighted gradient. The temperature T ranges from 2 to 5. The last hidden layer features of the model in the target task Mapping to the representation spaces of the cognitive and behavioral supervision layers using linear mapping or neural networks, respectively. ; The supervisor is trained using behavior supervision loss, distillation loss, and feature transfer loss for supervision fine-tuning. The training is based on cognitive supervision loss, distillation loss, and feature transfer loss, along with a supervisor for intention reasoning training.

10. The large model alignment system based on multi-layer cascaded dynamic supervision as described in claim 9, characterized in that, When training the supervisors for supervised fine-tuning / intent reasoning training, the weights of distillation loss and feature transfer loss are dynamically adjusted: the supervisory capabilities of the behavioral and cognitive supervisors are periodically evaluated on a standard alignment test set, and the weights of distillation loss and feature transfer loss are dynamically adjusted based on the evaluation results.

Images