Large language model alignment method and device based on neural signal driving

Through the large language model alignment method driven by neural signal, the concept-level control signals are extracted through the learning and fusion of positive and negative direction signals, the challenges of large language model in terms of security and reliability are solved, and higher alignment performance is achieved and controllability of behavior.

CN119067189AInactive Publication Date: 2024-12-03SHANDONG WOMENS UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411001313.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Large language models have challenges in terms of security and reliability, especially due to bias and discrimination caused by the diversity of learning corpus, as well as the 'big model hallucination' phenomenon, which leads to the model to generate misleading false information that affects its reliability and user-friendliness. sex.

Method used

Adopting a large language model alignment method based on neural signal-driven, by constructing positive and negative reaction samples, positioning and fusing positive and negative control signals, decoupling feature-independent signals, and extracting concept control signals, which are used to regulate the behavior of large language models. , achieve alignment.

Benefits of technology

Without affecting the original model parameters, the model alignment performance is improved, the behavior controllability and diversity and nature of generated content are enhanced, and the performance of large models on alignment tasks is significantly improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067189A_ABST
    Figure CN119067189A_ABST
Patent Text Reader

Abstract

The invention discloses a big language model alignment method and device based on neural signal driving, and the method comprises the steps: S1, constructing positive and negative reaction samples for a specific control task at a big language model training stage, and taking the positive and negative reaction samples as a comparison sample set; s2, guiding the generation direction of the large language model from two directions by comparing the sample set, and positioning and obtaining neural control signals which cause the large language model to generate different output behaviors, including a positive control signal and a negative control signal; s3, in a big language model reasoning stage, on the basis of linear subspace hypothesis, feature irrelevant signals are decoupled by fusing feature signals of the positive control signals and the negative control signals, and concept control signals are extracted; s4, using the concept control signal to control the large language model to realize nerve regulation of the large model; according to the method, the model alignment performance is effectively improved while the original generation capability of the model is maintained, and a new view angle and a new tool are provided for sustainable alignment and control of the large model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically to a large language model alignment method and device based on neural signal driving. Background Art

[0002] Currently, large language models (LLMs), such as ChatGPT, have made breakthrough progress in the fields of natural language processing (NLP) and artificial intelligence (AI) after being trained on large-scale data sets, and have been widely used in key fields such as academia, medicine, law, finance, and education.

[0003] However, with the rapid development of technology, the challenges of security and reliability of large language models have become increasingly prominent.

[0004] Values ​​and credibility are key factors in ensuring the security and quality of content generated by large language models (LLMs). Due to the diversity of the learning corpus, pre-trained LLMs may absorb and reflect bias and discrimination in the data. In addition, the so-called "large model hallucination" phenomenon may cause the model to generate misleading false information, thereby reducing its reliability.

[0005] In order to improve the user-friendliness and security of pre-trained LLMs, researchers have adopted methods including RLHF, RLAIF, RRHF, RAFT and DPO to achieve model alignment. However, these methods usually require fine-tuning of LLMs, rely on a large amount of carefully annotated data and powerful computing resources, and although LLMs may not directly exhibit bad behavior under ethical constraints, they already have such behavioral capabilities. The aligned large language model may still be bypassed by carefully crafted attack prompts.

[0006] Therefore, how to identify and control the key signals that affect the performance of large language models in order to improve the behavioral controllability of large models is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0007] In view of this, the present invention provides a large language model alignment method and device based on neural signal driving to solve some of the technical problems mentioned in the background technology.

[0008] In order to achieve the above object, the present invention adopts the following technical solution:

[0009] A large language model alignment method based on neural signal driving includes the following steps:

[0010] S1. During the large language model training phase, construct positive and negative response samples for specific control tasks as comparison sample sets;

[0011] S2. By comparing the sample sets, the large model is guided to generate directions from two directions, and the neural control signals that cause the large language model to produce different output behaviors are located and obtained, including positive control signals and negative control signals;

[0012] S3. In the large language model inference stage, based on the linear subspace hypothesis, the feature signals of positive control signals and negative control signals are fused to decouple feature-irrelevant signals and extract concept control signals;

[0013] S4. Use conceptual control signals to control the large language model and achieve neural regulation of the large model.

[0014] Preferably, in step S1, the same question and two answers of different polarities are used as a contrast stimulus sample in the form of question-answer pairs, thereby constructing a contrast sample set.

[0015] Preferably, the constructed comparison sample set is specifically:

[0016]

[0017] Where N is the number of samples, s i is the i-th contrast stimulus example;

[0018]

[0019] Among them, X i is the i-th specific prompt or question, is a positive response, representing the desired behavior, A negative response represents an undesirable behavior.

[0020] Preferably, the specific content of step S2 includes positive control signal learning and negative control signal learning; the positive control signal learning is specifically: for the positive control signal u, construct a training sample X i is the i-th specific prompt or question, For positive reply, use LoRA low-rank adaptation training to fine-tune the large language model to generate a low-rank matrix A u and B u , the attention parameters are updated using the following formula:

[0021]

[0022] Among them, u=(A u , B u ), Q, K, V are the parameters in the original large language model attention mechanism, Q′ u , K′ u , V′ u is the updated attention weight parameter of the original large language model, Au and B u A positive control signal that causes the large language model LLM to generate positive responses;

[0023] The specific learning of negative control signals is as follows: for negative control signals v, construct training samples For negative replies, use LoRA low-rank adaptation training to fine-tune the large language model to generate a low-rank matrix A υ and B υ , the attention parameters are updated using the following formula:

[0024]

[0025] Where v = (A υ , B υ ), Q, K, V are the parameters in the original large language model attention mechanism, Q′ υ , K′ υ , V′ υ is the updated attention weight parameter of the original large language model, A υ and B υ A negative control signal that causes the large language model (LLM) to generate negative responses.

[0026] Preferably, the specific content of the linear subspace assumption in step S3 is: high-level concepts are represented in a certain direction in the representation space. For counterfactual label pairs of concepts that only change in concept values, if the linear representation assumption holds, it indicates that the linear representation assumption holds for most concepts in the language model.

[0027] Preferably, in step S3, the method for obtaining the conceptual control signal is: subtracting the positive control signal from the negative control signal to obtain the conceptual control signal or vector, realizing the abstraction of the control signal from the specific to the general, decoupling the details, and realizing the driving of the model from the macroscopic conceptual direction:

[0028] w=uv=A u B u -A υ B υ

[0029] Among them, u=(A u , B u ), is the positive control signal, v=(A υ , B υ ), is a negative control signal, and W is a conceptual control signal.

[0030] Preferably, the specific content of step S4 is: using the concept control vector, adjusting the Q, K, and V matrices of the large language model to obtain:

[0031] Q″=Q+w (Q)

[0032] K″=K+w (K)

[0033] V″=V+w (V)

[0034] Where w = A u B u -A υ B u , Q″, K″, V″ are the adjusted model parameters, that is, the model parameters with the driving signal applied.

[0035] A large language model alignment device based on neural signal drive, based on the large language model alignment method based on neural signal drive, comprising: a control neural signal extraction module, a concept-level control signal construction module and a regulation module;

[0036] The control neural signal extraction module includes a comparison sample construction unit and a neural control signal acquisition unit;

[0037] A comparison sample construction unit is used to construct positive and negative response samples for a specific control task as a comparison sample set during the large language model training phase;

[0038] A neural control signal acquisition unit, used to guide the large model generation direction from two directions by comparing sample sets, locate and obtain neural control signals that cause the large language model to produce different output behaviors, including positive control signals and negative control signals;

[0039] The concept-level control signal building module is used to extract the concept control signal by fusing the feature signals of the positive control signal and the negative control signal, decoupling the feature-irrelevant signal, based on the linear subspace hypothesis, in the large language model inference stage;

[0040] The control module is used to use concept control signals to control the large language model and realize neural control of the large model.

[0041] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the neural signal-driven large language model alignment method.

[0042] A processing terminal includes a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, the large language model alignment method based on neural signal driving is implemented.

[0043] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a large language model alignment method and device based on neural signal driving, which can control the generation behavior of the model in a targeted manner without affecting the original model parameters, and better realize the alignment of the large language model, effectively improve the model alignment performance, and provide a new perspective and tool for the sustainable alignment and control of the large model; the present invention learns positive and negative direction control signals with very few learning samples, and further extracts high-level concept-level control signals through signal fusion, which can effectively guide the behavior of the model while maintaining the diversity and naturalness of the generated content; experiments on large language models of different sizes have proved that the present invention significantly improves the performance of large language models in various alignment tasks, reflecting good adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 A schematic diagram of a large language model alignment method based on neural signal driving provided by the present invention;

[0046] Figure 2 A schematic diagram showing the comparison of ASR indicators when different models provided by the present invention are configured in the Vanilla model, the RAIN method, and the present invention for processing high-risk inputs;

[0047] Figure 3 A schematic diagram comparing the harmful win rates of Vinilla provided by the present invention and the present invention;

[0048] Figure 4 A schematic diagram showing the comparison of the harmfulness winning rate between the RAIN model provided by the present invention and the present invention;

[0049] Figure 5 A schematic diagram showing the comparison of the effectiveness of the basic model Vanilla, the improved methods RAIN and TruthX, and the present invention in terms of response authenticity and informativeness indicators provided by the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The embodiment of the present invention discloses a large language model alignment method based on neural signal driving, such as Figure 1 , including the following steps:

[0052] S1. During the large language model training phase, construct positive and negative response samples for specific control tasks as comparison sample sets;

[0053] S2. By comparing the sample sets, the large model is guided to generate directions from two directions, and the neural control signals that cause the large language model to produce different output behaviors are located and obtained, including positive control signals and negative control signals;

[0054] S3. In the large language model inference stage, based on the linear subspace hypothesis, the feature signals of positive control signals and negative control signals are fused to decouple feature-irrelevant signals and extract concept control signals;

[0055] S4. Use conceptual control signals to control the large language model and achieve neural regulation of the large model.

[0056] In this embodiment, step S2 is performed while freezing the original large language model parameters.

[0057] In order to further implement the above technical solution, in step S1, the same question and two answers with different polarities are used as a contrast stimulus sample in the form of question-answer pairs, and then a contrast sample set is constructed.

[0058] In practical applications, given a prompt X i , large language model f θ The output for this prompt is O i , expressed as:

[0059]

[0060] Among them, θ is the weight parameter of the large language model f, P(·|·) is the probability of the large language model LLM generating the next token, |O i | is the length of LLM output, O i,<t All tokens generated before the tth token.

[0061] In order to further implement the above technical solution, the constructed comparison sample set is specifically:

[0062]

[0063] Among them, N is the number of samples, only a small number of samples are needed, s i is the i-th contrast stimulus example;

[0064]

[0065] Among them, X i is the i-th specific prompt or question, is a positive response, representing the desired behavior, A negative response represents an undesirable behavior.

[0066] In practical applications, most LLMs are composed of multiple stacked Transformer Block layers. The Attention module in the Block plays a vital role in the storage and retrieval of knowledge, and directly affects the response behavior of the LLM. Similarly, problems such as hallucinations in large models also originate from these Attention modules. Therefore, when looking for neural stimuli that affect changes in LLM behavior patterns, we mainly focus on the masked self-attention mechanism module in the Transformer Block.

[0067] In order to further implement the above technical solution, the specific content of step S2 includes positive control signal learning and negative control signal learning;

[0068] The specific learning of the forward control signal is as follows: for the forward control signal u, construct a training sample X i is the i-th specific prompt or question, For positive reply, use LoRA low-rank adaptation training to fine-tune the large language model to generate a low-rank matrix A u and B u , the attention parameters are updated using the following formula:

[0069]

[0070] Among them, u=(A u , B u ), Q, K, V are the parameters in the original large language model attention mechanism, Q′ u , K′ u , V′ u is the updated attention weight parameter of the original large language model, A u and B u A positive control signal that causes the large language model LLM to generate positive responses;

[0071] The specific learning of negative control signals is as follows: for negative control signals V, construct training samples For negative replies, use LoRA low-rank adaptation training to fine-tune the large language model to generate a low-rank matrix A υ and B υ , the attention parameters are updated using the following formula:

[0072]

[0073] Where v = (A υ , B υ ), Q, K, V are the parameters in the original large language model attention mechanism, Q′ υ , K′ υ , V′ υ is the updated attention weight parameter of the original large language model, A υ and B υ A negative control signal that causes the large language model (LLM) to generate negative responses.

[0074] In practical applications, directly using a control signal of a certain directional stimulus to influence the model, such as the positive control signal u, will introduce too much individual interference information into the model due to the particularity of each stimulus sample. Due to the deviation of individual samples, it will also affect the generalization ability of the model in different data distributions. For example, when you want to obtain emotional concept signals, you can extract the emotion of "happiness" from the sample of "exercise makes me happy", but at the same time it will also be mixed with interference information related to exercise "exercise".

[0075] In order to further implement the above technical solution, the specific content of the linear subspace hypothesis in step S3 is: high-level concepts are represented in a certain direction in the representation space. For example, concepts such as gender, emotion, or political inclination can be regarded as specific directions in the vector space. For counterfactual label pairs of concepts that only change in concept values, if the linear representation hypothesis holds, it means that the linear representation hypothesis holds for most concepts in the language model. By guiding the behavior of the large model through the concept control signal, the large model can have better OOD generalization capabilities.

[0076] In order to further implement the above technical solution, in step S3, the method for obtaining the conceptual control signal is: subtracting the positive control signal from the negative control signal to obtain the conceptual control signal or vector, realizing the abstraction of the control signal from the specific to the general, decoupling the details, and realizing the driving of the model from the macro-concept direction:

[0077] w=uv=A u B u -A υ B υ

[0078] Among them, u=(A u , B u ), is the positive control signal, v=(A υ , B υ ), is a negative control signal, and w is a conceptual control signal.

[0079] In order to further implement the above technical solution, the specific content of step S4 is: using the concept control vector, adjust the Q, K, V matrices of the large language model to obtain:

[0080] Q″=Q+w (Q)

[0081] K"=K+w (K)

[0082] V″=V+w (V)

[0083] Where w = A u B u -A υ B u , Q″, K″, V″ are the adjusted model parameters, that is, the model parameters with the driving signal applied.

[0084] After updating the large language model with the adjusted model parameters, output the result based on the input question or prompt:

[0085]

[0086] in, A new model controlled by specific neural signals. is the neural regulator, ΔW is the directional neural control signal generated by the neural regulator, For passing Model parameters after neural modulation.

[0087] In another embodiment, a comparative experiment was conducted:

[0088] (1) Tasks and datasets: We examine the effectiveness of our model using two tasks: harm-free generation and truthful generation. For the harm-free generation task, we use the HH dataset, which consists of sensitive questions that may cause potentially harmful responses from large language models. For the truthful content generation task, we use the TruthfulQA dataset, which is the most widely used benchmark for evaluating the authenticity of large language models and includes 817 questions from 38 categories.

[0089] (2) Comparison models: We use the current mainstream open source large models as references, including LLaMa-2-nochat (7B, 13B), LLaMA-2-chat (7B, 13B), LLaMA-3-nochat (8B), and LLaMA-3-Instruc (8B). We also select two representative alignment enhancement models, namely the RAIN model that controls the model response based on the inference stage, and the TruthX model that controls the model authenticity output by editing the internal representation of the large language model in the real space.

[0090] (3) Evaluation indicators: LLama Guard is used to evaluate the harmfulness of the content generated by the model, and ASR is used as an evaluation indicator to indicate how many of the sensitive questions caused the model to produce harmful outputs; for the evaluation of the authenticity and informativeness of the model response, two GPT-3 models are fine-tuned as classifiers for identifying the authenticity and informativeness of the response content. The percentage of authentic responses True (%), the percentage of informative responses Info (%), and True & Info (%) are calculated respectively.

[0091] (4) Experimental results:

[0092] In order to verify the performance of the DNS of the present invention in processing high-risk input, we used the ASR indicator to measure the probability of the model generating harmful output when facing sensitive questions. The experiment covered three model configurations, Llama-2-7B, Llama-2-13B and Llama-3-8B, and compared them with the Vanilla model, RAIN method and the method of the present invention. The experimental results are shown in the figure. Figure 2 shown.

[0093] Experimental data show that the DNS method is significantly better than the Vanilla and RAIN methods in all model configurations. Specifically, in the Llama-2-7B model, the DNS method reduces the ASR from 18.39% to 13.67%, while the ASR of the RAIN method is 16.00%. In the Llama-2-13B model, the DNS method reduces the ASR to 1.61%, which shows excellent harmful output control ability compared with 8.43% in the Vanilla mode and 6.15% in the RAIN method. For the Llama-3-8B model, the DNS method also performs well, reducing the ASR from 16.13% to 7.61%, which is a significant reduction compared with 12.53% of the RAIN method.

[0094] In order to further verify the advantages of the present invention over the baseline model, the DNS method of the present invention is compared with the Vinilla and RAIN models. Figure 3 and Figure 4shown.

[0095] Experimental results show that the neural signal-driven alignment method DNS of the present invention has significant advantages in improving the alignment performance of large language models LLMs; in comparison with original LLMs of different scales (LLaMA-2-7B, LLaMA-2-13B, LLaMA-3-8B), the winning rates of the DNS method are 97.60%, 87.76% and 90.38% respectively, which are significantly better than the original models, indicating that the DNS method exhibits higher alignment performance at all model scales; in addition, in comparison with the existing RAIN method, the winning rates of DNS on LLaMA-2-7B and LLaMA-2-13B are 92.96% and 62.50% respectively, further demonstrating the superiority of the DNS method. This result highlights the powerful ability of the DNS method in aligning model behaviors and values, and shows that by utilizing a small amount of control signals, DNS can achieve high controllability of the generation behavior of large language models without affecting the inherent knowledge of the model.

[0096] In order to verify the effectiveness of the present invention in responding to the authenticity and information indicators, the analysis covers LLMs of different sizes from LLaMA-2-7B-chat to LLaMA-2-13B-Chat, including the basic model Vanilla, the improved methods RAIN and TruthX, and the DNS method proposed in the present invention. The experimental results are shown in the figure. Figure 5 As shown, True means the answer is true, Info means the answer is informative, and True+Info means the answer is both true and informative.

[0097] Experiments show that the DNS method of the present invention improves the authenticity of the model output without losing the model informativeness in most configurations. In particular, in the Llama-2-7B-Chat configuration, the DNS method significantly improves the True&Info index to 81.22%, which shows that the method can effectively improve the authenticity of the output while maintaining high informativeness. In the Llama-2-13B-chat and Llama-3-8B-Instruct configurations, the DNS method still achieves a significant improvement in authenticity while maintaining no information loss, which further proves the universality and effectiveness of the DNS method. In the comparison with Llama-2-13B-chat, although the informativeness is slightly lower than that of RAIN, it has an execution efficiency similar to that of Vanilla.

[0098] In this example, a neural signal-driven large language model alignment method (DNS) of the present invention extracts concept-level control signals by learning positive and negative direction signals, so that the alignment of the large language model is achieved with only a small number of positive and negative stimulus samples; DNS can be used as a plug-and-play plug-in. When in the inference stage, the large language model with frozen parameters can generate responses consistent with human preferences. At the same time, DNS also helps to improve the performance of well-aligned models such as LLaMA-2-chat; experimental results show that DNS effectively improves the model alignment performance and provides a new perspective and tool for the sustainable alignment and control of large models.

[0099] A large language model alignment device based on neural signal drive, based on a large language model alignment method based on neural signal drive, comprising: a control neural signal extraction module, a concept-level control signal construction module and a regulation module;

[0100] The control neural signal extraction module includes a comparison sample construction unit and a neural control signal acquisition unit;

[0101] A comparison sample construction unit is used to construct positive and negative response samples for a specific control task as a comparison sample set during the large language model training phase;

[0102] A neural control signal acquisition unit, used to guide the large model generation direction from two directions by comparing sample sets, locate and obtain neural control signals that cause the large language model to produce different output behaviors, including positive control signals and negative control signals;

[0103] The concept-level control signal building module is used to extract the concept control signal by fusing the feature signals of the positive control signal and the negative control signal, decoupling the feature-irrelevant signal, based on the linear subspace hypothesis, in the large language model inference stage;

[0104] The control module is used to use concept control signals to control the large language model and realize neural control of the large model.

[0105] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a large language model alignment method based on neural signal driving.

[0106] A processing terminal includes a memory and a processor. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, a large language model alignment method based on neural signal driving is implemented.

[0107] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0108] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large language model alignment method based on neural signal driving, characterized in that: The following steps are involved: S1. During the large language model training phase, construct positive and negative response samples for specific control tasks as comparison sample sets; S2. By comparing the sample sets, the large model is guided to generate directions from two directions, and the neural control signals that cause the large language model to produce different output behaviors are located and obtained, including positive control signals and negative control signals; S3. In the large language model inference stage, based on the linear subspace hypothesis, the feature signals of positive control signals and negative control signals are fused to decouple feature-irrelevant signals and extract concept control signals; S4. Use conceptual control signals to control the large language model and achieve neural regulation of the large model.

2. According to the neural signal driven large language model alignment method of claim 1, it is characterized in that: In step S1, in the form of question-answer pairs, the same question and two answers with different polarities are used as a contrast stimulus sample to construct a contrast sample set.

3. According to the neural signal driven large language model alignment method of claim 1, it is characterized in that: The constructed comparison sample set is specifically: Where N is the number of samples, s i is the i-th contrast stimulus example; Among them, X i is the i-th specific prompt or question, is a positive response, representing the desired behavior, is a negative response, representing an undesirable behavior.

4. According to the neural signal driven large language model alignment method of claim 1, it is characterized in that: The specific content of step S2 includes positive control signal learning and negative control signal learning; The specific learning of the forward control signal is as follows: for the forward control signal u, construct a training sample X i is the i-th specific prompt or question, For positive reply, use LoRA low-rank adaptation training to fine-tune the large language model to generate a low-rank matrix A u and B u , the attention parameters are updated using the following formula: Among them, u=(A u , B u ), Q, K, V are the parameters in the original large language model attention mechanism, Q′ u , K′ u , V′ u is the updated attention weight parameter of the original large language model, A u and B u A positive control signal that causes the large language model LLM to generate positive responses; The specific learning of negative control signals is as follows: for negative control signals V, construct training samples For negative replies, use LoRA low-rank adaptation training to fine-tune the large language model to generate a low-rank matrix A υ and B υ , the attention parameters are updated using the following formula: Where v = (A υ , B υ ), Q, K, V are the parameters in the original large language model attention mechanism, Q′ υ , K′ υ 、V υ ′ is the updated attention weight parameter of the original large language model, A υ and B υ A negative control signal that causes the large language model (LLM) to generate negative responses.

5. The neural signal driven large language model alignment method according to claim 1, characterized in that: The specific content of the linear subspace hypothesis in step S3 is: high-level concepts are represented in a certain direction in the representation space. For counterfactual label pairs of concepts that only change in concept values, if the linear representation hypothesis holds, it means that the linear representation hypothesis holds for most concepts in the language model.

6. The neural signal driven large language model alignment method according to claim 1, characterized in that: In step S3, the method for obtaining the conceptual control signal is: subtracting the positive control signal from the negative control signal to obtain the conceptual control signal or vector, realizing the abstraction of the control signal from the specific to the general, decoupling the details, and realizing the driving of the model from the macro-concept direction: w=u-v=A u B u -A υ B υ Among them, u=(A u , B u ), is the positive control signal, v=(A υ , B υ ), is a negative control signal, and W is a conceptual control signal.

7. The neural signal driven large language model alignment method according to claim 6, characterized in that: The specific content of step S4 is: using the concept control vector, adjust the Q, K, V matrices of the large language model to obtain: Q″=Q+w (Q) K″=K+w (K) V"=V+w (V) Where w = A u B u -A υ B u , Q″, K″, V″ are the adjusted model parameters, that is, the model parameters with the driving signal applied.

8. A large language model alignment device driven by neural signals, characterized in that: A neural signal driven large language model alignment method based on any one of claims 1 to 7, comprising: a control neural signal extraction module, a concept-level control signal construction module and a regulation module; The control neural signal extraction module includes a comparison sample construction unit and a neural control signal acquisition unit; A comparison sample construction unit is used to construct positive and negative response samples for a specific control task as a comparison sample set during the large language model training phase; A neural control signal acquisition unit, used to guide the large model generation direction from two directions by comparing sample sets, locate and obtain neural control signals that cause the large language model to produce different output behaviors, including positive control signals and negative control signals; The concept-level control signal building module is used to extract the concept control signal by fusing the feature signals of the positive control signal and the negative control signal, decoupling the feature-irrelevant signal, based on the linear subspace hypothesis, in the large language model inference stage; The control module is used to use concept control signals to control the large language model and realize neural control of the large model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the large language model alignment method based on neural signal driving is implemented as described in any one of claims 1 to 7.

10. A processing terminal, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the large language model alignment method based on neural signal driving as described in any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Large model defense method based on semantic logic

    CN119378613A