Long dialogue user behavior analysis method based on large language model

Abstract

The invention discloses a long dialogue user behavior analysis method based on a large language model. Comprising a user behavior analysis large model module, a long dialogue embedding compression memory module and a domain-specific retrieval enhancement fine tuning module. The user behavior analysis large model module analyzes long dialogues between users based on a large language model, and studies and judges behaviors of the users; a long dialogue embedding compression memory module is introduced into a large language model based on a Transform architecture, and a mask local attention and long-term linear attention mechanism is combined, so that a user behavior analysis large model module can process a long dialogue with limited memory and computing resources; and the domain specific retrieval enhancement fine tuning module is generated by combining instruction fine tuning and retrieval enhancement and is used for improving the user behavior analysis capability of the large language model in a specific legal domain.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence data analysis, and in particular to a method for analyzing user behavior in long conversations based on a large language model. Background Art

[0002] With the advancement of artificial intelligence and natural language processing technologies, long conversation analysis is becoming increasingly important in fields such as electronic forensics, data analysis, and behavioral analysis. However, existing methods face challenges with memory and computing resources when processing long conversations, especially when real-time analysis is required. Furthermore, existing methods have limited capabilities for analyzing user behavior in specific domains, making it difficult to accurately understand users' behavioral intentions and emotional tendencies. Traditional Transformer architectures suffer from quadratic complexity when processing long sequences, limiting their application in long conversation analysis. Furthermore, existing methods lack adaptability in specific domains, making it difficult to accurately analyze user behavior.

[0003] Traditional natural language processing methods typically use models based on the Transformer architecture when processing long conversations. The Transformer's self-attention mechanism enables parallel processing of input sequences, significantly improving training and inference efficiency. However, standard Transformer models also face challenges with memory and computational resources when processing long sequences. Because the time and space complexity of its self-attention mechanism are both quadratic, processing long conversation data consumes significant amounts of memory and computational resources. In addition to memory and computational resource limitations, existing methods also lack adaptability to specific domains. In practical applications, long conversation analysis often requires optimization for specific domains, such as legal, medical, or internal corporate document processing. In these domains, traditional models may not be able to accurately understand conversation content and user behavior. Therefore, adapting general language models to specific domains and improving their performance in these areas is an important research direction. Summary of the Invention

[0004] To address these issues, this paper discloses a method for analyzing long-conversation user behavior based on a large language model. The method comprises a large user behavior analysis model module, a domain-specific retrieval enhancement and fine-tuning module, and a long-conversation embedding and compression memory module. Through the synergistic effect of these modules, the present invention efficiently processes long-conversation data and improves the accuracy and robustness of user behavior analysis in specific legal fields.

[0005] The present invention proposes a method for analyzing user behavior in long conversations based on a large language model, aiming to efficiently process long conversations and enhance analytical capabilities in specific legal fields. The method first utilizes a large user behavior analysis model module to conduct an in-depth analysis of long conversations between users. This module is based on Tongyi Qianwen's 7B model and can accurately analyze user behavior patterns. In order to optimize the processing efficiency of long conversations, the present invention introduces a long conversation embedded compressed memory module, which combines masked local attention and long-term linear attention mechanisms, enabling the large user behavior analysis model module to efficiently process long conversation content with limited memory and computing resources. In addition, the present invention also applies a domain-specific retrieval enhancement fine-tuning module. By combining instruction fine-tuning and retrieval enhancement generation technology, the large language model's user behavior analysis capabilities in specific legal fields are significantly enhanced, thereby providing legal professionals with a more accurate and efficient analysis tool.

[0006] The user behavior analysis large model module is based on Tongyi Qianwen's 7B model. Its specific implementation steps are as follows: First, input processing is performed, taking the user's long conversation data as input. The input data can be textual conversation records or preprocessed structured data. The preprocessing step includes text cleaning and multimodal data textualization to ensure the quality of the input data. Next, user behavior analysis and assessment are performed, using Tongyi Qianwen's 7B model to analyze the input long conversation and determine whether the user has engaged in illegal activities in specific areas. The input long conversation is segmented into multiple paragraphs, each of which serves as an independent input unit, and a corresponding feature vector is generated. The model uses a multi-layer Transformer architecture and a self-attention mechanism to capture contextual information within the paragraphs. The feature vectors of all paragraphs are aggregated to form a global feature representation of the entire long conversation. Based on this global feature representation of the long conversation, the model generates an answer based on the context of the long conversation, directly indicating whether the user has engaged in illegal activities and providing corresponding explanations and evidence. Finally, output generation is performed, and based on the assessment results, a user behavior analysis report is generated. The report contains detailed analysis results of the user's behavior, specifically the assessment of illegal activities in specific areas. The report is displayed and output in text form or visual chart form to facilitate user understanding and use.

[0007] Tongyi Qianwen's 7B model is a large language model based on a multi-layer Transformer architecture, containing 7 billion parameters, capable of handling complex language tasks and generating natural language text. During the pre-training phase, the model is trained using large-scale, diverse text data to learn common language patterns and knowledge, thereby possessing extensive language understanding and generation capabilities. Through a domain-specific retrieval enhancement fine-tuning module, the model is fine-tuned and retrieval-enhanced on a dataset in a specific domain, enabling it to accurately understand and analyze conversation content in a specific domain and accurately determine whether a user has engaged in illegal behavior. In addition, the model introduces a long conversation embedding compressed memory module, combined with masked local attention and long-term linear attention mechanisms, capable of processing extremely long input sequences with limited memory and computing resources, providing rich contextual information for behavioral analysis. Tongyi Qianwen's 7B model is configured as an analysis and judgment model that can generate detailed answers based on the input of long conversation content, provide analysis and judgment on user behavior, and provide basic data support for user behavior analysis.

[0008] Furthermore, the long conversation embedding compressed memory module is introduced to handle long conversations and expand context. This module uses local attention to capture fine-grained dependencies and compressed memory to model long-range connections. This achieves excellent compression by only adding a fixed amount of memory parameters to the Transformer. This can directly replace the standard multi-head attention mechanism, supporting long-context expansion and fine-tuning of existing large user behavior analysis model modules.

[0009] Specifically:

[0010] (1) Input segmentation processing: The input sequence is divided into segments X of length N s , each segment is processed independently, but cross-segment information transmission is achieved through compressed memory. The formula is:

[0011]

[0012] (2) Local attention calculation: for the current segment X s Calculate the standard scaled dot product attention to generate the local context A dot , where W Q , W K , W V Is the projection matrix, which maps the input to the query, key, and value space Q s , K s , V s :

[0013] Q s =X s W Q , K s =X s WK , V s =X s W V

[0014]

[0015] (3) Memory retrieval: using the current query Q s From memory M s-1 Retrieve long-range context A mem :

[0016]

[0017] Where σ is a nonlinear activation function used to ensure numerical stability, z s-1 is a normalization term that records the cumulative sum of historical key vectors.

[0018] (1) Memory update: Use the association matrix update rule to compress the key-value pairs of the current segment into memory:

[0019] M s ←M s-1 +σ(K s ) T V s

[0020]

[0021] (2) Context fusion: Dynamically fuse local and long-range context via a learnable gating scalar β:

[0022] A=sigmoid(β)⊙A mem +(1-sigmoid(β))⊙A dot

[0023] β is a learnable scalar (independent for each head) mapped to [0,1] by sigmoid, dynamically weighing the local A dot and long-range A mem Importance of context. ⊙ represents the Hadamard product, which mixes the two contexts and outputs A.

[0024] (3) Multi-head output: H heads output A 1 ,...,A H Splicing along the feature dimension, W O Mapping the result back to the model dimension gives the final output:

[0025] O s =[A 1 ;...;A H ]W O

[0026] Furthermore, by introducing the long conversation embedding compression memory module to process long conversations and expand the context, the training and inference optimization of this module are specifically as follows:

[0027] (1) Segmented back propagation: memory state M s and z s As a hidden variable, the gradient is calculated by time expansion during training, and combined with gradient checkpoints to save memory.

[0028] (2) Position encoding strategy: Local attention uses standard position encoding, while the retrieval and update of compressed memory do not rely on position encoding and only capture global semantics.

[0029] (3) Streaming reasoning: The memory size is a fixed value related to the Transformer, which supports infinite input. The memory is updated after each segment is processed, and the original key-value cache is discarded. The complexity is reduced from O(N 2 ) is reduced to O(N).

[0030] Furthermore, the domain-specific retrieval enhancement fine-tuning module is introduced to enhance the large language model's ability to analyze user behavior in specific legal fields. The domain-specific retrieval enhancement fine-tuning module fine-tunes the pre-trained large language model, enabling it to effectively distinguish relevant documents from interference documents and generate answers containing chained reasoning based on relevant documents. Its characteristics are:

[0031] (1) Training data construction: The training model ignores interference documents and extracts answers only from relevant documents. Given question Q and relevant documents D * and interference documents D1,D2,...,D k , the training data is divided into two categories:

[0032] Samples containing relevant documents accounting for P%:

[0033] Q+D * +D1+D2+...+D k →A *

[0034] A * Based on D * The generated answer contains the * The text fragment is directly quoted in the question and the CoT process of explaining how the answer is derived from the quotation is used.

[0035] Percentage (1-P%) of samples containing relevant documents:

[0036] Q+D1+D2+...+D k →A *

[0037] To simulate retrieval failure scenarios, the model needs to rely on its own knowledge to generate answers.

[0038] (2) Training Objective: By adding Q&A predictions in specific legal fields and supervising fine-tuning to minimize the following losses, the user behavior analysis model module can be specialized in specific legal fields, while preventing the model from being overly dependent on context and retaining a certain degree of domain understanding:

[0039]

[0040] Where N is the number of training samples, the model needs to generate an answer A that includes references and reasoning * .

[0041] (3) Reasoning phase: The model receives question Q and retrieved Top-k documents and generates an answer:

[0042] A test =LLM(Q+D1+D2+...+D k )

[0043] Based on the filtering capabilities learned during training, the model can identify relevant documents and generate answers with chained reasoning. This chained reasoning requires that answers include direct quotations from the document and logical reasoning, which improves the interpretability of answers and prevents the model from outputting only brief answers.

[0044] Beneficial effects of the present invention:

[0045] (1) Based on the user behavior analysis model module, it conducts in-depth analysis of user behavior in long conversations and can accurately identify potential illegal activities. It supports segmented processing and streaming reasoning, taking into account both real-time and scalability.

[0046] (2) By embedding a compressed memory module into a long conversation and combining it with masked local attention and long-term linear attention mechanisms, the memory and computing resource consumption are significantly reduced, enabling large user behavior analysis models to efficiently process ultra-long conversations while maintaining contextual coherence and semantic integrity.

[0047] (3) By utilizing the domain-specific retrieval enhancement fine-tuning module, combined with instruction fine-tuning and retrieval enhancement generation technology, the professional judgment ability of the user behavior analysis model in the legal field is greatly improved. It can accurately identify illegal behaviors in user conversations and provide explainable chain reasoning analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 、 one Flowchart of a method for analyzing long conversation user behavior based on a large language model. DETAILED DESCRIPTION

[0049] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are intended only to illustrate the present invention and are not intended to limit the scope of the present invention. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, and the terms "inward" and "outward" refer to directions toward or away from the geometric center of a particular component, respectively.

[0050] like Figure 1 As shown, this embodiment is a long conversation user behavior analysis method based on a large language model,

[0051] The following steps are involved:

[0052] Step 1: Analyze long conversations between users using the user behavior analysis model module. The user behavior analysis model module analyzes user behavior based on Tongyi Qianwen's 7B model.

[0053] Step 2: Introducing a long conversation embedding compressed memory module, combined with masked local attention and long-term linear attention mechanisms, enables the large user behavior analysis model module to process long conversations with limited memory and computing resources;

[0054] Step 3: Apply the domain-specific retrieval enhancement fine-tuning module to improve the large language model's ability to analyze user behavior in specific legal fields by combining instruction fine-tuning and retrieval enhancement generation. The domain-specific retrieval enhancement fine-tuning module enables the model to be fine-tuned and perform retrieval enhancement on datasets in specific fields, thereby enabling accurate understanding and analysis of conversation content in specific fields.

[0055] Furthermore, we introduce the long conversation embedding compressed memory module to handle long conversations and expand context. This module uses local attention to capture fine-grained dependencies and compressed memory to model long-range connections. This achieves superior compression by only adding a fixed amount of memory parameters associated with the Transformer. This can directly replace the standard multi-head attention mechanism, supporting long-context expansion and fine-tuning of existing large user behavior analysis model modules. Specifically:

[0056] (1) Input segmentation processing: The input sequence is divided into segments X of length N s , each segment is processed independently, but cross-segment information transmission is achieved through compressed memory. The formula is:

[0057]

[0058] (2) Local attention calculation: for the current segment X s Calculate the standard scaled dot product attention to generate the local context A dot , where W Q , W K , WV Is the projection matrix, which maps the input to the query, key, and value space Q s , K s , V s :

[0059] Q s =X s W Q , K s =X s W K , V s =X s W V

[0060]

[0061] (3) Memory retrieval: using the current query Q s From memory M s-1 Retrieve long-range context A mem :

[0062]

[0063] Where σ is a nonlinear activation function used to ensure numerical stability, z s-1 is a normalization term that records the cumulative sum of historical key vectors.

[0064] (1) Memory update: Use the association matrix update rule to compress the key-value pairs of the current segment into memory:

[0065] M s ←M s-1 +σ(K s ) T V s

[0066]

[0067] (2) Context fusion: Dynamically fuse local and long-range context via a learnable gating scalar β:

[0068] A=sigmoid(β)⊙A mem +(1-sigmoid(β))⊙A dot

[0069] β is a learnable scalar (independent for each head) mapped to [0,1] by sigmoid, dynamically weighing the local A dot and long-range A mem Importance of context. ⊙ represents the Hadamard product, which mixes the two contexts and outputs A.

[0070] (3) Multi-head output: H heads of output A 1 ,..., A H Concatenate along feature dimension, W O Map the result back to model dimension to get the final output:

[0071] O s = [A 1 ;... ; A H ]W O

[0072] Further, by introducing the long dialogue embedding compression memory module to process long dialogues and expand the context, the training and inference optimization of the module is as follows:

[0073] (1) Segmented backpropagation: memory state M s and z s as hidden variables, the gradient is calculated by time expansion during training, combined with gradient checkpoint to save memory.

[0074] (2) Position encoding strategy: local attention uses standard position encoding, while retrieval and update of compressed memory do not depend on position encoding, only capturing global semantics

[0075] (3) Streaming inference: the memory size is a fixed value related to the Transformer, supporting infinite long input, updating the memory after processing each segment and discarding the original key-value cache. The complexity is reduced from O(N 2 ) to O(N).

[0076] Further, by introducing the domain-specific retrieval enhancement fine-tuning module to improve the ability of large language models to analyze user behavior in specific legal domains, the domain-specific retrieval enhancement fine-tuning module fine-tunes the pre-trained large language model to effectively distinguish relevant documents and interference documents, and generates answers containing chain reasoning based on relevant documents, its features are:

[0077] (1) Training data construction: the training model ignores interference documents and only extracts answers from relevant documents. Given a question Q, relevant documents D * and interference documents D1, D2,..., D k , the training data is divided into two categories:

[0078] P% of relevant document containing samples:

[0079] Q+D * +D1+D2+...+D k →A *

[0080] A * is the answer generated based on D * , which contains chain reasoning from D* The text fragment is directly quoted in the question and the CoT process of explaining how the answer is derived from the quotation is used.

[0081] The percentage (1-P%) of samples containing relevant documents:

[0082] Q+D1+D2+...+D k →A *

[0083] To simulate retrieval failure scenarios, the model needs to rely on its own knowledge to generate answers.

[0084] (2) Training Objective: By adding Q&A predictions in specific legal fields and supervising fine-tuning to minimize the following losses, the user behavior analysis model module can be specialized in specific legal fields, while preventing the model from being overly dependent on context and retaining a certain degree of domain understanding:

[0085]

[0086] Where N is the number of training samples, the model needs to generate an answer A that includes references and reasoning * .

[0087] (3) Reasoning phase: The model receives question Q and retrieved Top-k documents and generates an answer:

[0088] A test =LLM(Q+D1+D2+...+D k )

[0089] Based on the filtering capabilities learned during training, the model can identify relevant documents and generate answers with chained reasoning. This chained reasoning requires that answers include direct quotations from the document and logical reasoning. This improves the interpretability of answers and prevents the model from outputting only brief answers. This improves performance in specific domains by approximately 29.73%.

[0090] The technical means disclosed in the solution of the present invention are not limited to the technical means disclosed in the above-mentioned embodiment, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A method for analyzing user behavior in long conversations based on a large language model, characterized in that: The following steps are involved: Step 1: Analyze long conversations between users using the user behavior analysis model module. The user behavior analysis model module analyzes user behavior based on Tongyi Qianwen's 7B model. Step 2: Introducing a long conversation embedding compressed memory module, combined with masked local attention and long-term linear attention mechanisms, enables the large user behavior analysis model module to process long conversations with limited memory and computing resources; Step 3: Apply the domain-specific retrieval enhancement fine-tuning module to improve the user behavior analysis capabilities of the large language model in specific legal fields by combining instruction fine-tuning and retrieval enhancement generation.

2. The method for analyzing user behavior in long conversations based on a large language model according to claim 1, characterized in that: The user behavior analysis model module in step 1 is based on the 7B model of Tongyi Qianwen. The specific implementation steps are as follows: Step 11: Input processing: The user's long conversation data is used as input. The input data is text-based conversation records or pre-processed structured data. The pre-processing steps include text cleaning and multimodal data textualization to ensure the quality of the input data. Step 12: User Behavior Analysis and Judgment: Utilize Tongyi Qianwen's 7B model to analyze the input long conversation and determine whether the user has committed illegal acts in specific areas. The input long conversation is divided into multiple paragraphs, with each paragraph acting as an independent input unit to generate a corresponding feature vector. The user behavior analysis large model module uses a multi-layer Transformer architecture and a self-attention mechanism to capture contextual information within the paragraphs. It aggregates the feature vectors of all paragraphs to form a global feature representation of the entire long conversation. Based on the global feature representation of the long conversation, the user behavior analysis large model module generates an answer based on the context of the long conversation. The answer from the user behavior analysis large model module will directly indicate whether the user has committed illegal acts and provide corresponding explanations and evidence. Step 13: Output generation: Based on the analysis results, a user behavior analysis report is generated. The report contains detailed analysis results of user behavior, especially the analysis results of illegal behavior in specific areas; The report is displayed and output in text form or visual chart form to facilitate user understanding and use.

3. The method for analyzing user behavior in long conversations based on a large language model according to claim 1, characterized in that: Step 2 is to process the long conversation and expand the context by introducing the long conversation into the compression memory module; the specific steps are: Step 21: Input segmentation processing: The input sequence is divided into segments X of length N s , each segment is processed independently, but cross-segment information transmission is achieved through compressed memory. The formula is: Step 22: Local attention calculation: for the current segment X s Calculate the standard scaled dot product attention to generate the local context A dot , where W Q , W K , W V Is the projection matrix, which maps the input to the query, key, and value space Q s , K s , V s : Q s =X s W Q ,K s =X s W K ,V s =X s W V Step 23: Memory Retrieval: Using Current Query Q s From memory M s-1 Retrieve long-range context A mem : Where σ is a nonlinear activation function used to ensure numerical stability, z s-1 is a normalization term that records the cumulative sum of historical key vectors; Step 24: Memory update: Use the association matrix update rule to compress the key-value pairs of the current segment into memory: M s ←M s-1 +σ(K s ) T V s Step 25: Context Fusion: Dynamically fuse local and long-range context via a learnable gating scalar β: A=sigmoid(β)⊙A mem +(1-sigmoid(β))⊙A dot β is a learnable scalar that is independently mapped to [0,1] by sigmoid, dynamically weighing the local A dot and long-range A mem The importance of context; ⊙ represents the Hadamard product, and the output is A after mixing the two contexts; Step 26: Multi-head Output: Output A of H Heads 1 ,...,A H Splicing along the feature dimension, W O Mapping the result back to the model dimension gives the final output: O s =[A 1 ;...;A H ]W O。 4. The method for analyzing user behavior in long conversations based on a large language model according to claim 3, characterized in that: By introducing the long conversation embedding compressed memory module to process long conversations and expand context, the training and inference optimization of this module are specifically as follows: (1) Segmented back propagation: memory state M s and z s As a hidden variable, the gradient is calculated by time expansion during training, and combined with gradient checkpoints to save memory; (2) Position encoding strategy: local attention uses standard position encoding, while the retrieval and update of compressed memory do not rely on position encoding and only capture global semantics; (3) Streaming reasoning: The memory size is a fixed value related to the Transformer, which supports infinite input. The memory is updated after each segment is processed, and the original key-value cache is discarded. The complexity is reduced from O(N 2 ) is reduced to O(N).

5. The method for analyzing user behavior in long conversations based on a large language model according to claim 1, characterized in that: The domain-specific retrieval enhancement fine-tuning module in step 3 fine-tunes the pre-trained large language model to enable it to effectively distinguish relevant documents from interference documents and generate answers containing chained reasoning based on relevant documents. The specific steps are as follows: Step 31: Training data construction: The training model ignores interference documents and extracts answers only from relevant documents. Given question Q, relevant documents D * and interference documents D1,D2,...,D k , the training data is divided into two categories: Samples containing relevant documents accounting for P%: Q+D * +D1+D2+...D k →A * A * Based on D * The generated answer contains the * A directly quoted text fragment and an explanation of the CoT process of how to derive the answer from the quote; The percentage (1-P%) of samples containing relevant documents: Q+D1+D2+...+D k →A * To simulate retrieval failure scenarios, the model needs to rely on its own knowledge to generate answers. Step 32: Training Objective: By adding Q&A predictions in specific legal fields, supervise and fine-tune the user behavior analysis model module to make it specialized in specific legal fields, prevent it from being overly dependent on context, and retain a certain degree of domain understanding. Where N is the number of training samples, the model needs to generate an answer A that includes references and reasoning * ; Step 33: Reasoning phase: The model receives question Q and retrieved Top-k documents and generates an answer: A test =LLM(Q+D1+D2+...+D k ) Based on the screening ability learned by the model during training, the model identifies relevant documents and generates answers with chain reasoning with citations; based on chain reasoning, the answer must include content directly quoted from the document and the logical reasoning process to improve the explainability of the answer.

Images