Multi-context session large language model system

Abstract

The invention relates to the field of data processing, and relates to a multi-context session big language model system, which comprises a multi-role context management module used for converting a dialogue history into a structured sequence containing role identifiers, role background information, dialogue contents, a site, preferences and timestamps; the role information injection module is used for extracting role metadata and dialogue content from the structured sequence and performing structural fusion on the role metadata and the dialogue content through a predefined injection mechanism so as to construct a large language model input prompt containing explicit role features; and the role perception reasoning generation module is internally provided with a large language model based on a Transform architecture and is used for dynamically adjusting attention weight according to target role characteristics carried in the input prompt so as to generate a reply consistent with a target role standing field. When the system is used for processing the multi-party dialogue, the reply which is highly consistent with the role site and is logically coherent can be generated, and the intention understanding accuracy is high.

Description

Technical Field

[0001] This invention relates to the field of data processing. More specifically, this invention relates to a multi-context conversational large language model system. Background Technology

[0002] With the advancement of deep learning technology, large language models based on the Transformer architecture have achieved remarkable results in text generation and multi-turn dialogue tasks, and have been widely applied in scenarios such as intelligent customer service, virtual character interaction, and meeting assistance.

[0003] However, existing technologies still face significant challenges when processing conversational data involving multiple participants and complex contexts. Current large language models primarily rely on standard self-attention mechanisms to handle long text sequences, treating the entire dialogue history as originating from a single, anonymous or uniform user. All dialogue rounds are simply concatenated chronologically into a linear text sequence as the model's contextual input; the model lacks built-in mechanisms to identify and differentiate between different participants in the dialogue. Therefore, existing large language models process de-identified text streams, failing to bind and understand content with specific speakers. While this algorithm can capture semantic relationships between tokens, it essentially treats the input data as a homogeneous text stream, lacking explicit modeling and differentiation of the high-level semantic feature of "role identity."

[0004] In multi-role scenarios, this indiscriminate attention calculation method has significant drawbacks: it struggles to structurally distinguish the evolving stances and viewpoints of different speakers, making the model susceptible to interference from irrelevant role information during inference. Due to the lack of an attention-focusing mechanism targeting specific roles, the model cannot dynamically adjust its attention weight to the context based on the currently defined role, leading to the following technical problems frequently encountered in the generated responses: 1. Role confusion and conflicting viewpoints: The model cannot distinguish the viewpoints of different roles. For example, in a debate, it treats both sides' arguments as contradictory statements from the same user, potentially generating a response that attempts to reconcile the contradictions but is vague and confused, failing to reflect the adversarial nature of the debate.

[0005] 2. Intent Interpretation Bias: The model may incorrectly correlate the context of questions or statements made by different roles. For example, it may mistake a hypothetical question raised by role A as a fact confirmed by role B, leading to a fundamental bias in the understanding of the current query.

[0006] 3. Lack of specificity and direction in responses: Because it cannot identify the audience and participants in the conversation, the model's responses are generalized and indiscriminate, failing to provide personalized responses for specific roles (e.g., "Regarding the request just made by user A, my suggestion is...").

[0007] 4. Not suitable for complex collaborative scenarios: In advanced scenarios that require accurate recording of speaker attribution, summarizing multiple viewpoints, or role-playing (such as mock interviews or courtroom debates), existing technologies are completely inadequate. Summary of the Invention

[0008] To address the technical problems of role confusion, conflicting viewpoints, and biased understanding of intent that arise when existing large language models process conversational data with multiple participants and complex contexts, this invention provides solutions in the following aspects.

[0009] In a first aspect, the present invention provides a multi-context conversational large language model system, comprising: a multi-role context management module, which serves as the input end of the system, for receiving raw multi-party dialogue data, assigning a unique role identifier to each participant, and converting the dialogue history into a structured sequence containing role identifiers, role background information, dialogue content, stance, preferences and timestamps, so as to output standardized multi-role context data; The role information injection module is used to extract role metadata and dialogue content from the structured sequence, and to structurally fuse the role metadata and dialogue content through a predefined injection mechanism, thereby constructing a large language model input prompt containing explicit role features. The role-aware reasoning generation module has a built-in large language model based on the Transformer architecture. It is used to receive the input prompts and perform reasoning. During the reasoning process, it uses an improved attention calculation unit to dynamically adjust the attention weight according to the target role features carried in the input prompts, thereby generating a response that is consistent with the position of the target role.

[0010] Preferably, the improved attention calculation unit employs a conditional attention mechanism, specifically including a role gating subunit; the role gating subunit is used to calculate the correlation between each token in the input sequence and the current target role, generating a gating scalar; the role perception reasoning generation module uses the gating scalar to perform a weighted adjustment on the original attention weights, the adjustment formula being: ; In the formula, For standard attention weights, For a gated scalar targeting role r, This represents the final attention weight.

[0011] Preferably, the character gating subunit includes a neural network; the neural network is configured to receive the character embedding vector of the target character. and the representation vector of the current token As input, the gating scalar is calculated to output a value between 0 and 1 to suppress the weight of irrelevant role information in attention calculation.

[0012] Preferably, the role-aware reasoning generation module further includes a role embedding unit; the role embedding unit is used to assign a learnable role embedding vector to each role, and add the role embedding vector to the corresponding word embedding vector at the input layer of the model as a comprehensive input representation to the attention calculation unit.

[0013] Preferably, the role information injection module employs a structured prompting construction algorithm; this algorithm converts the structured sequence into a JSON-like hierarchical structure format and introduces a dedicated special token in the vocabulary, the special token including at least a tag for identifying the start of a role and a tag for identifying attribute key values; the large language model is configured to parse and distinguish dialogue content from role metadata by recognizing the special token.

[0014] Preferably, the role-aware reasoning generation module is further configured with a structure type embedding unit and a hierarchical position encoding unit; the structure type embedding unit is used to assign a type ID to each token in the input prompt to distinguish whether it belongs to plain text, object key or object value; the hierarchical position encoding unit is used to encode the absolute position of the token in the sequence as well as the depth and sibling node order in the JSON-like hierarchical structure to represent the logical subordinate relationship of the role attributes.

[0015] Preferably, the multi-head attention mechanism in the role perception reasoning generation module includes a specific role modeling head; the specific role modeling head is a specially defined attention head used to specifically capture and model the interaction relationships and position conflicts between different roles, rather than only focusing on the semantic association of the text.

[0016] Preferably, it also includes a model fine-tuning module for supervised fine-tuning of the large language model based on an augmented dataset; the augmented dataset contains data generated through mirror dialogue and perspective switching strategies: the mirror dialogue strategy generates dialogue responses set by two different roles for the same situation; the perspective switching strategy requires the model to restate or comment on a neutral narrative in the tone of different roles.

[0017] Preferably, it also includes an interactive control interface; the interface is used to receive control commands input by the user, the commands including defining the relationship between roles or specifying the target role identity for generating a response; the role-aware reasoning generation module responds to the control commands, adjusts the target role conditions in the reasoning process, and generates content with a specific orientation.

[0018] Preferably, each record in the structured sequence stored by the multi-role context management module is a tuple; the tuple is in the form of <role identifier, dialogue content, timestamp>, and the system tracks and distinguishes the evolution of different roles' viewpoints in a long context based on the tuple.

[0019] The beneficial effects of this invention are as follows: Addressing the problems of role confusion, inconsistent stances, and biased intent understanding that existing large language models easily encounter when processing multi-party dialogues, this invention constructs a complete system encompassing multi-role context management, information injection, and perceptual reasoning. This system achieves standardized processing of multi-party dialogue data and explicit role feature fusion. Compared to traditional methods that rely solely on text prompts to distinguish roles, this invention utilizes an improved attention computing unit to dynamically adjust the weights in the reasoning process based on the target role's features. This allows the model to more accurately focus on information related to the current target role, thereby generating responses that are highly consistent with the role's stance and logically coherent, effectively improving the intent understanding accuracy of large language models. Attached Figure Description

[0020] Figure 1 This is a schematic diagram illustrating the structure of a multi-context conversational large language model system according to an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0023] Implementation example of a multi-context conversational large language model system: like Figure 1 As shown, the multi-context conversational large language model system of the present invention includes: The multi-role context management module, as the input end of the system, is used to receive raw multi-party dialogue data, assign a unique role identifier to each participant, and transform the dialogue history into a structured sequence containing role identifiers, role background information, dialogue content, stance, preferences and timestamps, so as to output standardized multi-role context data. The role information injection module is used to extract role metadata and dialogue content from the structured sequence, and to structurally fuse the role metadata and dialogue content through a predefined injection mechanism, thereby constructing a large language model input prompt containing explicit role features. The role-aware reasoning generation module has a built-in large language model based on the Transformer architecture. It is used to receive the input prompts and perform reasoning. During the reasoning process, it uses an improved attention calculation unit to dynamically adjust the attention weight according to the target role features carried in the input prompts, thereby generating a response that is consistent with the position of the target role.

[0024] In this embodiment, the multi-role context management module is used to perform role registration and assignment, as well as structured sequence storage. The role registration and assignment process includes: when a new session starts or a new participant joins, the system assigns a unique role identifier (Role ID), such as Role_A, Role_B.

[0025] Character background information typically includes basic identity attributes, personality and language style, as well as knowledge boundaries and abilities.

[0026] The basic identity attributes are used to define the most fundamental labels of "who I am," ensuring that the model does not confuse roles. These include social attributes and physical / physiological characteristics.

[0027] Social attributes include: occupation (e.g., doctor, lawyer, primary school student), age, gender, and educational background.

[0028] Physical / physiological characteristics include: health status, physical characteristics, etc.

[0029] Personality and language style define "how I speak," influencing the word choice and tone of the text generated by the model. Personality and language style include personality traits and language habits.

[0030] Personality traits include: introverted / extroverted, rational / emotional, optimistic / pessimistic, impatient / gentle.

[0031] Language style includes: favorite catchphrases, dialects, and level of formality.

[0032] Knowledge boundaries and capabilities are used to define "what I know" and prevent characters from "breaking out of character." Knowledge boundaries and capabilities include areas of expertise and information asymmetry.

[0033] A professional field is used to define the specific knowledge that a character is proficient in.

[0034] Information asymmetry is used to limit the information that a character knows and does not know in the current conversation.

[0035] Character metadata can include: character background information, character identifier, character ID, character type, character preferences, and stance.

[0036] The system in this embodiment constructs a multi-role context management framework. This framework assigns and maintains a unique, identifiable role identifier for each participant within a single dialogue session. Through an innovative role information injection mechanism, when organizing multi-turn dialogue history into input prompts for a Large Language Model (LLM), the speaker's role metadata (such as role ID, role type, role preferences, and stance) is structurally integrated with the speech content.

[0037] This allows LLM to establish explicit associations between discourse and speaker identity during its attention calculation process, thereby achieving contextual understanding and content generation based on role perception. Ultimately, the system can clearly distinguish and track the viewpoints, positions, and intentions of different roles, generating highly consistent and accurate responses that can be tailored to specific roles or integrate all roles.

[0038] The system implemented in this embodiment fundamentally solves the problem of identity confusion in multi-role dialogues, ensuring that the generated responses are logically consistent in terms of viewpoints and positions, and accurately reflect the dynamics of the dialogue. Furthermore, by establishing a strong "role-content" correlation, the model can gain a deeper understanding of the semantic context of the dialogue, the intentions of each participant, and the interactive relationships between viewpoints, thereby making more accurate inferences and responses.

[0039] In one embodiment, the improved attention calculation unit employs a conditional attention mechanism, specifically including a role gating subunit. The role gating subunit calculates the correlation between each token in the input sequence and the current target role, generating a gating scalar. The role perception reasoning generation module uses the gating scalar to weight and adjust the original attention weights, using the following formula: ; In the formula, For standard attention weights, For a gated scalar targeting role r, This represents the final attention weight.

[0040] Standard attention weights The attention given to Token i to Token j was measured. This embodiment adds a gating scalar to this. This is used to evaluate whether Token i is related to the currently generated target role r.

[0041] By introducing a conditional attention mechanism and a role-gated subunit, the relevance of each token in the input sequence to the current target role can be quantitatively calculated. Unlike traditional self-attention mechanisms that treat all contextual information equally, this invention uses a generated gating scalar to weight and adjust the original attention weights, effectively suppressing interference information unrelated to the target role and significantly enhancing the model's ability to capture specific role features in complex contexts.

[0042] In one embodiment, the character gating subunit includes a neural network configured to receive the character embedding vector of the target character. and the representation vector of the current token As input, the gating scalar is calculated to output a value between 0 and 1 to suppress the weight of irrelevant role information in attention calculation.

[0043] By configuring a neural network to specifically implement the role-gating subunit, and utilizing the interactive computation of the role embedding vector and the token representation vector, a finely tuned gating scalar with values ​​between 0 and 1 is output. This allows the system to flexibly control the information flow in a non-linear manner, effectively "soft-shielding" irrelevant role information at the attention computation level, further improving the purity and role-specificity of the model-generated content.

[0044] In one embodiment, the role-aware reasoning generation module further includes a role embedding unit; the role embedding unit is used to assign a learnable role embedding vector to each role, and add the role embedding vector to the corresponding word embedding vector at the input layer of the model as a comprehensive input representation to the attention calculation unit.

[0045] The method in this embodiment enables the input representation to carry identity features in the initial stage.

[0046] By introducing learnable role embedding vectors into the model input layer and adding them to word embedding vectors, role identity information is directly mapped into a high-dimensional semantic space. Compared to identifying roles solely through names at the text level, this low-level feature fusion approach allows the model to perceive the role attributes associated with each lexical unit, thereby maintaining the stability of role identity in deep semantic understanding and reducing the likelihood of the model "forgetting" the currently played role.

[0047] In one embodiment, the role information injection module employs a structured prompting construction algorithm; the algorithm converts the structured sequence into a JSON-like hierarchical structure format and introduces a dedicated special token in the vocabulary, the special token including at least a tag for identifying the start of a role and a tag for identifying attribute key values; the large language model is configured to parse and distinguish dialogue content from role metadata by recognizing the special token.

[0048] A structured prompting algorithm is used to transform linear dialogues into a JSON-like hierarchical structure, and a special token is introduced. Compared with the traditional plain text concatenation method, this structured input clearly defines the boundary between dialogue content and character metadata, greatly reducing the ambiguity when the model parses complex contexts, and enabling the model to more accurately identify the subordinate relationship between character attributes and dialogue content.

[0049] In one embodiment, the role-aware reasoning generation module is further configured with a structure type embedding unit and a hierarchical position encoding unit; the structure type embedding unit is used to assign a type ID to each token in the input prompt to distinguish whether it belongs to plain text, object key or object value; the hierarchical position encoding unit is used to encode the absolute position of the token in the sequence as well as the depth and sibling node order in the JSON-like hierarchical structure to represent the logical subordinate relationship of the role attributes.

[0050] In addition to conventional token embedding and position embedding, this embodiment also introduces structure type embedding. Each token in the sequence is assigned a type ID to identify whether it belongs to "object key", "object value", or "plain text".

[0051] To enable the model to understand the JSON tree structure, hierarchical encoding is implemented. This encodes not only the position of the token in the linear sequence but also its depth and sibling index within the logical tree, thus helping the model understand attribute relationships. For example, understanding that "Personality: Introverted" is an attribute belonging to "Role A".

[0052] By configuring structure type embedding units and hierarchical positional encoding units, the model not only distinguishes the types of text, keys, and values, but also encodes node depth and sibling order. This allows the model to overcome the limitations of traditional linear sequence encoding, understand the logical hierarchy and subordinate structure of role attributes in the dialogue tree, and thus maintain a clear logical reasoning chain when dealing with complex nested dialogues or intertwined discussions involving multiple people.

[0053] In one embodiment, the multi-head attention mechanism in the role-aware reasoning generation module includes a specific role modeling head; the specific role modeling head is a specially defined attention head used to specifically capture and model the interaction relationships and positional conflicts between different roles, rather than focusing only on textual semantic associations.

[0054] In a multi-head attention mechanism, a specific role modeling head is designated to capture the interaction relationships and positional conflicts between roles. Compared with traditional attention mechanisms that only focus on the semantic association of text, this invention can explicitly model social dynamic features such as "who opposes whom" or "who supports whom," so that the generated responses are not only semantically fluent, but also accurately reflect the interpersonal relationships and emotional tensions in the dialogue.

[0055] In one embodiment, the system further includes a model fine-tuning module for performing data augmentation on the multi-role dialogue dataset to obtain an augmented dataset, and performing supervised fine-tuning on the large language model based on the augmented dataset. The data augmentation methods for the multi-role dialogue dataset include mirror dialogue strategy and perspective switching strategy. The mirror dialogue strategy is to generate dialogue responses set by two different roles for the same situation. The perspective switching strategy is to require the model to repeat or comment on a neutral narrative in the tone of different roles.

[0056] For the same scenario, two dialogue responses with different role settings are generated; for example, for the scenario of "project failure," responses from both an "optimistic leader" and a "pessimistic analyst" are generated simultaneously. This forces the model to learn to distinguish subtle role characteristics.

[0057] The perspective-switching strategy can efficiently generate data from different roles' perspectives on the same event, enhancing the model's understanding of the roles' positions.

[0058] In this embodiment, supervised fine-tuning of the large language model enhances its ability to understand and utilize "role-content" pairs, shaping its attentional patterns for role perception. This maximizes the consistency between the model output and the role setting, rather than just its relevance to the question.

[0059] Supervised fine-tuning using augmented datasets based on mirror dialogue and perspective-switching strategies forces the model to learn the expressive differences of different roles within the same context. This approach effectively overcomes the problem of scarce training data for specific roles, significantly improving the model's ability to flexibly switch between different role settings and the accuracy of restating or commenting on neutral facts in a specific tone.

[0060] In one embodiment, an interactive control interface is also included; the interface is used to receive control instructions input by the user, the instructions including defining the relationship between roles or specifying the target role identity for generating a response; the role-aware reasoning generation module responds to the control instructions, adjusts the target role conditions in the reasoning process, and generates content with a specific orientation.

[0061] For example, the relationship between roles can be defined as "doctor" and "patient". The target role for generating the response can be specified as: "Please summarize as the moderator" or "Please refute user A's point of view".

[0062] This embodiment provides an interactive control interface that allows users to dynamically define role relationships or specify target identities. This changes the traditional passive model generation mode, giving users the ability to intervene in the generation process. It enables the model to dynamically adjust inference conditions in response to external instructions, generating content with specific targeting and customized features, thus enhancing the system's interactivity and application flexibility.

[0063] In one embodiment, each record in the structured sequence stored by the multi-role context management module is a tuple; the tuple is in the form of <role identifier, dialogue content, timestamp>, and the system tracks and distinguishes the evolution of different roles' viewpoints in a long context based on the tuple.

[0064] The structured sequence is stored in a tuple format that includes role identifiers, dialogue content, and timestamps. This strict data structure ensures that the source and timing of each dialogue record can be accurately traced in long context scenarios, effectively preventing the loss of tracking or misattribution of viewpoint evolution, which is common in long dialogues, and ensuring the consistency of role viewpoints over long periods of time.

[0065] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.

Claims

1. A multi-context conversational large language model system, characterized in that, include: The multi-role context management module, as the input end of the system, is used to receive raw multi-party dialogue data, assign a unique role identifier to each participant, and transform the dialogue history into a structured sequence containing role identifiers, role background information, dialogue content, stance, preferences and timestamps, so as to output standardized multi-role context data. The role information injection module is used to extract role metadata and dialogue content from the structured sequence, and to structurally fuse the role metadata and dialogue content through a predefined injection mechanism, thereby constructing a large language model input prompt containing explicit role features. The role-aware reasoning generation module has a built-in large language model based on the Transformer architecture. It is used to receive the input prompts and perform reasoning. During the reasoning process, it uses an improved attention calculation unit to dynamically adjust the attention weight according to the target role features carried in the input prompts, thereby generating a response that is consistent with the position of the target role.

2. The multi-context conversational large language model system as described in claim 1, characterized in that, The improved attention calculation unit employs a conditional attention mechanism, specifically including a role-gating subunit. This subunit calculates the correlation between each token in the input sequence and the current target role, generating a gating scalar. The role perception reasoning generation module uses this gating scalar to weight and adjust the original attention weights using the following formula: ; In the formula, For standard attention weights, For a gated scalar targeting the role r, This represents the final attention weight.

3. The multi-context conversational large language model system as described in claim 2, characterized in that, The role gating subunit includes a neural network; this neural network is configured to receive the role embedding vector of the target role. and the representation vector of the current token As input, the gating scalar is calculated to output a value between 0 and 1 to suppress the weight of irrelevant role information in attention calculation.

4. The multi-context conversational large language model system as described in claim 1, characterized in that, The role perception reasoning generation module also includes a role embedding unit; the role embedding unit is used to assign a learnable role embedding vector to each role, and add the role embedding vector to the corresponding word embedding vector at the input layer of the model as a comprehensive input representation to the attention calculation unit.

5. The multi-context conversational large language model system as described in claim 1, characterized in that, The character information injection module employs a structured prompting construction algorithm; this algorithm converts the structured sequence into a JSON-like hierarchical structure format and introduces a special token in the vocabulary, which includes at least a tag for identifying the start of a character and a tag for identifying attribute key values; The large language model is configured to parse and distinguish between dialogue content and role metadata by recognizing the special token.

6. The multi-context conversational large language model system as described in claim 5, characterized in that, The role perception reasoning generation module is also configured with a structure type embedding unit and a hierarchical position encoding unit; the structure type embedding unit is used to assign a type ID to each token in the input prompt to distinguish whether it belongs to ordinary text, object key or object value; The hierarchical position encoding unit is used to encode the absolute position of the Token in the sequence, as well as its depth and sibling node order in the JSON-like hierarchical structure, to represent the logical subordinate relationship of the role attribute.

7. The multi-context conversational large language model system as described in claim 1, characterized in that, The multi-head attention mechanism in the role perception reasoning generation module includes a specific role modeling head; the specific role modeling head is a specially defined attention head used to specifically capture and model the interaction relationships and position conflicts between different roles, rather than just focusing on the semantic association of the text.

8. The multi-context conversational large language model system as described in claim 1, characterized in that, It also includes a model fine-tuning module for supervised fine-tuning of the large language model based on an augmented dataset; the augmented dataset contains data generated through mirror dialogue and perspective switching strategies: the mirror dialogue strategy generates dialogue responses set by two different roles for the same situation; the perspective switching strategy requires the model to restate or comment on a neutral narrative in the tone of different roles.

9. The multi-context conversational large language model system as described in claim 1, characterized in that, It also includes an interactive control interface; the interface is used to receive control commands input by the user, the commands including defining the relationship between roles or specifying the target role identity for generating a response; The role perception reasoning generation module responds to the control command, adjusts the target role conditions in the reasoning process, and generates content with a specific orientation.

10. The multi-context conversational large language model system as described in any one of claims 1 to 9, characterized in that, Each record in the structured sequence stored by the multi-role context management module is a tuple; the tuple is in the form of <role identifier, dialogue content, timestamp>, and the system tracks and distinguishes the evolution of different roles' viewpoints in a long context based on this tuple.

Images