A large language model training data generation method based on traffic playback and implicit feedback

Abstract

This invention relates to the field of computer technology and provides a method for generating training data for a large language model based on traffic replay and implicit feedback. The method includes: acquiring interactive session data between users and the large language model in online services, including user questions, model answers, and subsequent user behaviors; identifying user preference signals based on the interactive session data using a hierarchical implicit feedback judgment algorithm; prioritizing high-confidence implicit signals in the hierarchical implicit feedback judgment algorithm, including collaborative learning recognition based on user editing behavior, recognition based on user query reconstruction, and automated judgment based on confidence scores; generating a structured preference training dataset based on the user preference signals, including multiple preference pairs, each pair including user questions, correct answers, and incorrect answers; effectively improving the authenticity and quality of the data, alleviating reward model cracking and model conformity problems, and achieving continuous optimization and rapid iteration of model performance.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for generating training data for large language models based on traffic replay and implicit feedback. Background Technology

[0002] In the alignment phase of large language model training, reinforcement learning techniques based on human feedback are currently widely used. This method first requires human annotators to label multiple responses generated from the same prompt word with their preferences, distinguishing between high-quality and low-quality responses. Then, an independent reward model is trained using the labeled preference data, enabling it to score the quality of any prompt-response pair. Finally, the pre-trained language model is used as the policy in reinforcement learning, with the output of the reward model serving as the reward signal. Fine-tuning is then performed using algorithms such as proximal policy optimization, gradually aligning the model's output policy with human preferences.

[0003] However, reinforcement learning techniques based on human feedback rely heavily on manual preference labeling, resulting in extremely high time and financial costs. Furthermore, the labeling process needs to be restarted with each model iteration, leading to poor scalability. Secondly, differing subjective judgment standards among labelers introduce unavoidable subjectivity and inconsistency, potentially even leading to malicious labeling. During training, the model may learn to deceive the reward model, generating answers that cater to superficial human preferences rather than objectively correct ones to obtain high scores, resulting in so-called reward-cheating. Additionally, the process requires simultaneously maintaining and training both the policy and reward models, leading to high algorithmic complexity. Reinforcement learning training itself is unstable, sensitive to hyperparameters, and difficult to debug. Moreover, models over-optimized for higher rewards may output seemingly authoritative but actually incorrect answers, generating misleading content.

[0004] In view of this, a method for generating training data for large language models based on traffic replay and implicit feedback is proposed. Summary of the Invention

[0005] This invention provides a method for generating training data for large language models based on traffic replay and implicit feedback, which addresses the problems of high time and cost costs and poor scalability of existing models.

[0006] This invention provides a method for generating training data for large language models based on traffic replay and implicit feedback, comprising: Acquire user interaction session data with the large language model in the online service, the interaction session data including user questions, model answers, and user subsequent behaviors; Based on the interactive session data, user preference signals are identified through a hierarchical implicit feedback judgment algorithm; wherein, the hierarchical implicit feedback judgment algorithm prioritizes processing implicit signals with high confidence, including collaborative learning recognition based on user editing behavior, recognition based on user query reconstruction, and automated judgment based on confidence scores. A structured preference training dataset is generated based on the user preference signal. The preference training dataset includes multiple preference pairs, each of which includes a user question, a correct answer, and an incorrect answer.

[0007] Furthermore, the step of identifying user preference signals based on the interactive session data through a hierarchical implicit feedback judgment algorithm includes: Based on a preset confidence priority order, the following steps are executed sequentially: collaborative learning recognition based on user editing behavior, recognition based on user query reconstruction, and automated judgment based on confidence scores. Specifically, the collaborative learning recognition based on user editing behavior is used to process explicit user editing signals with the highest confidence, the recognition based on user query reconstruction is used to process implicit user feedback signals with medium confidence, and the automated judgment based on confidence scores is used to process fuzzy interaction signals with low confidence.

[0008] Furthermore, the hierarchical implicit feedback judgment algorithm employs a dynamic progressive processing mechanism, wherein: The collaborative learning recognition based on user editing behavior is processed as the first level. When substantial editing of the model's answer by the user is detected, the highest priority preference pair is generated. The recognition based on user query reconstruction is a second-level process, which is initiated only when no valid signal is recognized in the first level. It identifies user query reconstruction behavior through semantic similarity analysis. The automated judgment based on confidence score is the third level of processing, which is activated only when no valid signal is identified in the first two levels, and is performed automatically by the referee model.

[0009] Furthermore, the collaborative learning recognition based on user editing behavior is processed as the first level. When substantial editing of the model's response by the user is detected, a highest-priority preference pair is generated, including: Detect user editing behavior on the model's answer by comparing the model's original answer with the text that the user finally inputs or confirms after the model's answer; The edit distance algorithm is used to calculate the edit distance between the original answer of the model and the edited text by the user, and minor changes are filtered based on a preset edit distance threshold; When the edit distance exceeds the preset edit distance threshold, a preference pair is automatically generated; wherein the user-edited text is considered the correct answer, the original model answer is considered the incorrect answer, and the preference pair is assigned the highest confidence level.

[0010] Furthermore, the step of calculating the edit distance between the model's original answer and the user-edited text using the edit distance algorithm includes: The Levenstein distance algorithm is used to calculate the minimum number of single-character editing operations between two texts. The system determines whether a user's editing behavior has substantial corrective significance based on a preset edit distance threshold.

[0011] Furthermore, the identification based on user query reconstruction is a second-level process, initiated only when no valid signal is identified in the first level. It identifies user query reconstruction behavior through semantic similarity analysis, including: Extract two consecutive user questions from an interactive session, and use a text embedding model pre-trained based on contrastive learning to generate semantic vectors for the two user questions. The text embedding model is used to optimize and train for the user query intent recognition task. Calculate the semantic similarity score between the two semantic vectors, including performing multi-dimensional feature fusion processing on the semantic vectors; The semantic similarity score is compared with a preset similarity threshold. When the semantic similarity score exceeds the similarity threshold, it is determined that the user has reconstructed the query. When it is determined that the user has restructured the query, the large language model's answer to the previous user's question is marked as an incorrect answer, and a preference pair to be improved is generated, which includes the current user's question, the correct answer to be determined, and the incorrect answer.

[0012] Furthermore, the multi-dimensional feature fusion processing of the semantic vector includes: The attention mechanism employing query reconstruction dynamically adjusts attention weights based on the time interval and edit distance of user queries; where: The time interval weight is calculated based on the timestamp difference between two consecutive user questions, with a higher weight for shorter time intervals. The edit distance weight is calculated based on the user's text editing behavior in the current session; the greater the edit distance, the higher the weight. The time interval weight and edit distance weight are fused into the attention calculation of the semantic vector to generate a weighted semantic similarity score.

[0013] Furthermore, the automated judgment based on confidence scoring is a third-level process, initiated only when no valid signal is identified in the first two levels. Automated judgment is performed using a referee model, including: The dialogue context, which includes user questions and responses from a large language model, is input into the judge model, which is a high-performance language model. The referee model generates a better answer than the original answer of the large language model as a candidate correct answer, and at the same time outputs a confidence score for the generated result. Based on the comparison between the confidence score and the preset confidence threshold, a decision is made on whether to adopt the generated preference pair.

[0014] Furthermore, the output includes a confidence score for the generated result, comprising: The referee model performs a multi-dimensional quality assessment on the candidate correct answers it generates, and generates a numerical confidence score based on the results of the multi-dimensional quality assessment. The confidence score is used to quantify the degree of certainty that the referee model has regarding the quality of the generated answers.

[0015] Furthermore, the step of generating a structured preference training dataset based on the user preference signal includes: User preference signals from different levels are uniformly formatted to generate data items that conform to the triplet structure. Based on the confidence level of each data item, data items with a confidence level higher than a preset threshold are selected and retained. The filtered data items are integrated into a structured dataset for training the direct preference optimization algorithm.

[0016] As can be seen from the above technical solutions, the present invention has the following advantages: This invention captures high-confidence implicit signals through collaborative learning recognition and query reconstruction based on user editing behavior, ensuring that the training data originates from real-world application scenarios. This significantly improves the authenticity and quality of the data and effectively avoids the subjective bias and inconsistency problems caused by traditional manual annotation. Simultaneously, by combining an automated evaluation method based on confidence scoring, a complete structured preference training dataset is constructed. This not only effectively reduces annotation costs and algorithm complexity but also alleviates reward model cracking and model conformity issues by directly generating data that conforms to the training format of large language models. Ultimately, this achieves continuous optimization and rapid iteration of model performance. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of an embodiment of a method for generating training data for a large language model based on traffic replay and implicit feedback in this invention. Figure 2 This is a schematic diagram of the first-level processing in this invention; Figure 3 This is a schematic diagram of the second-level processing in this invention; Figure 4 This is a flowchart illustrating the third-level processing in this invention; Figure 5 This is a flowchart illustrating the step of generating a structured preference training dataset in this invention. Detailed Implementation

[0018] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0019] Example 1 The implementation method in this embodiment can be implemented in a system, on a server, or on a terminal; no specific limitation is made. The method in this application will be described below from the perspective of system implementation. Please refer to... Figure 1 The method provided in this application includes the following steps: S1. Obtain the interaction session data between the user and the large language model in the online service. The interaction session data includes user questions, model answers, and subsequent user behaviors. In this embodiment, the system captures the complete dialogue process between the user and the intelligent customer service system in real time by deploying a traffic log module on the online service front end. When a user accesses the intelligent customer service platform, the system automatically creates an independent session identifier and begins recording all interactive data in that session. The specifically recorded interactive session data includes three core parts: first, each question asked by the user, such as how to set up an email client; second, each answer generated by the system based on a large language model, including the specific operation steps provided by the model; and finally, the user's subsequent behavior after receiving the model's answer. These behaviors are obtained by monitoring operations on the user interface, including direct modification of the model's answer text, repeated questioning of the same question, copying the answer content, or directly closing the session window.

[0020] For example, in a real-world application scenario, a user asks how to implement the quicksort algorithm in Python, and the model generates an answer containing specific code. When the user uses this code, they discover an error, modify the code directly in the editable dialog interface, and resubmit it. At this point, the system will fully record the original question, the model's initial answer, the user's final modified version, the timestamp of the modification operation, and the session identifier.

[0021] These interactive data are transmitted to the data storage center via a real-time streaming pipeline. Each data unit contains complete time-series information and metadata, ensuring that subsequent processing can accurately reconstruct the complete context of each session. The data acquisition process is implemented using a distributed architecture, which can support the massive data acquisition needs in high-concurrency scenarios, providing a sufficient data foundation for subsequent implicit feedback analysis.

[0022] S2. Based on interactive session data, user preference signals are identified through a hierarchical implicit feedback judgment algorithm; wherein, the hierarchical implicit feedback judgment algorithm prioritizes processing implicit signals with high confidence, including collaborative learning recognition based on user editing behavior, recognition based on user query reconstruction, and automated judgment based on confidence scores. This step includes the following: Based on a preset confidence priority order, the following steps are executed sequentially: collaborative learning recognition based on user editing behavior, recognition based on user query reconstruction, and automated judgment based on confidence scores. Among them, collaborative learning recognition based on user editing behavior is used to process explicit user editing signals with the highest confidence, recognition based on user query reconstruction is used to process implicit user feedback signals with medium confidence, and automated judgment based on confidence scores is used to process fuzzy interaction signals with low confidence.

[0023] Specifically, the system uses an interaction analysis module to intelligently analyze the collected interaction session data. This module employs a preset confidence level priority order to construct a three-level progressive analysis system. Specifically, the system first attempts to extract the highest-confidence explicit editing signals from user behavior. If no valid signals are found, it proceeds to implicit feedback analysis with medium confidence. Finally, for ambiguous interactions that remain uncertain, an automated evaluation mechanism is activated.

[0024] In this embodiment, the hierarchical implicit feedback judgment algorithm adopts a dynamic progressive processing mechanism, wherein: S21. Collaborative learning recognition based on user editing behavior is processed as the first level. When substantial editing of the model's answer by the user is detected, the highest priority preference pair is generated. Please see Figure 2 This step includes the following sub-steps: S211. Detect user editing behavior on the model's answer by comparing the original model answer with the text that the user finally inputs or confirms after the model's answer; S212. The edit distance algorithm is used to calculate the edit distance between the original answer of the model and the edited text by the user, and minor changes are filtered based on a preset edit distance threshold; The calculation of edit distance using the edit distance algorithm includes the following sub-steps: 1. Use the Levenstein distance algorithm to calculate the minimum number of single-character editing operations required between two texts; 2. Determine whether the user's editing behavior has substantial corrective significance based on a preset editing distance threshold.

[0025] S213. When the edit distance exceeds the preset edit distance threshold, a preference pair is automatically generated; wherein, the user-edited text is taken as the correct answer, the original model answer is taken as the incorrect answer, and the preference pair is assigned the highest confidence level.

[0026] Specifically, the system detects editing behavior by comparing the last response generated by the model in a session with the text ultimately entered or confirmed by the user after that response. Specifically, it monitors the user's final submission in an editable dialog box; when it detects that the user has modified and resubmitted the model's original response, it identifies it as editing behavior. Secondly, the system uses the Levenstein distance algorithm to calculate the edit distance between the model's original response and the user's edited text. This algorithm quantifies the degree of text difference by calculating the minimum number of single-character editing operations required between the two texts, including insertion, deletion, and replacement operations. The system presets an edit distance threshold of five characters. This threshold is based on statistical analysis of a large number of user editing behaviors and can effectively distinguish between substantial content modifications and simple spelling corrections. When the edit distance is greater than or equal to five characters, the system determines that the user's editing behavior has substantial corrective significance.

[0027] In real-world applications, for example, a user asks how to write a Python bubble sort algorithm, and the model generates an answer containing specific code. The user then discovers a logical error in the code and directly modifies the core algorithm in the front-end edit box, changing the number of characters to twenty. Upon detecting this editing behavior, the system calculates and confirms that the edit distance exceeds a preset threshold. It then automatically generates a preference pair, where the user's correct code is the preferred answer, and the original incorrect code from the model is the inferior answer. This preference pair is assigned the highest confidence level of 1.0.

[0028] S22. User query reconstruction identification is processed as the second level, and is initiated only when no valid signal is identified in the first level. User query reconstruction behavior is identified through semantic similarity analysis. Please see Figure 3 This step includes the following sub-steps: S221. Extract two consecutive user questions from the interactive session, and use a text embedding model pre-trained based on contrastive learning to generate semantic vectors for the two user questions. The text embedding model is used to optimize the training for the user query intent recognition task. S222. Calculate the semantic similarity score between two semantic vectors, including multi-dimensional feature fusion processing of the semantic vectors; The query reconstruction attention mechanism dynamically adjusts attention weights based on the time interval and edit distance of user queries. Specifically, the time interval weight is calculated based on the difference in timestamps between two consecutive user queries, with a higher weight for shorter time intervals. The edit distance weight is calculated based on the user's text editing behavior in the current session, with a higher weight for larger edit distances. The time interval weight and edit distance weight are then integrated into the attention calculation of semantic vectors to generate a weighted semantic similarity score.

[0029] S223. Compare the semantic similarity score with a preset similarity threshold. When the semantic similarity score exceeds the similarity threshold, it is determined that the user has reconstructed the query. S224. When it is determined that the user has restructured the query, mark the large language model's answer to the previous user's question as an incorrect answer, and generate a preference pair to be improved that includes the current user's question, the correct answer to be determined, and the incorrect answer.

[0030] Specifically, the system first extracts two consecutive user questions from the interactive session. For example, a user might first ask how to learn Python, and then shortly after, ask how to start learning Python programming. To accurately understand the user's intent, the system uses a text embedding model pre-trained based on contrastive learning to generate semantic vectors for these two user questions. This text embedding model is trained through self-supervised learning and is specifically optimized for the user query intent recognition task. Its training process brings the vector representations of semantically similar queries closer together and pushes away the vector representations of dissimilar queries, enabling the model to deeply capture the semantic essence of the query, rather than just matching surface words.

[0031] Secondly, the system employs a dedicated attention mechanism for query reconstruction, which dynamically adjusts attention weights based on the time interval and edit distance of user queries. The time interval weight is obtained by calculating the difference in timestamps between two consecutive user queries; the shorter the time interval, the higher the weight, as repeated queries within a short period often indicate user dissatisfaction with the previous answer. The edit distance weight is calculated by analyzing the user's text editing behavior in the current session; the larger the edit distance, the higher the weight, reflecting the degree of user correction to the model's output. The system integrates these two weights into the attention calculation of semantic vectors, giving weighted emphasis to key semantic features, ultimately generating a more accurate weighted semantic similarity score.

[0032] In real-world applications, for example, a user first asks for information about tourist attractions in Guangxi. After the model answers, the user asks again within 30 seconds for information about famous landmarks in Guangxi. The system generates semantic vectors for both questions using a text embedding model, calculating an initial similarity score of 0.88. This is then weighted using time interval and edit distance weights, raising the final similarity score to 0.93. The system compares this score to a preset similarity threshold of 0.90. Since the score exceeds the threshold, it determines that the user has performed query reconstruction. This threshold is set based on statistical analysis of a large amount of user session data and can effectively distinguish between genuine query reconstruction and ordinary topic shifts.

[0033] When the system determines that a user has restructured their query, it marks the previous user's answer to the large language model as an incorrect answer and generates a preference pair to be improved, containing the current user's question, the correct answer to be determined, and the marked incorrect answer. This preference pair to be improved is then passed to the next processing stage, where the referee model generates the correct answer, thus completing the construction of the preference pair.

[0034] S23. Automated judgment based on confidence score is the third level of processing. It is initiated only when no valid signal is identified in the first two levels, and automated judgment is performed through the referee model.

[0035] Please see Figure 4 This step includes the following sub-steps: S231. Input the dialogue context, which includes user questions and large language model answers, into the judge model, which is a high-performance language model; S232. Generate a candidate correct answer that is better than the original answer of the large language model through the referee model, and output the confidence score of the generated result at the same time. Specifically, the referee model performs multi-dimensional quality assessment on the candidate correct answers it generates, and generates a numerical confidence score based on the multi-dimensional quality assessment results. The confidence score is used to quantify the degree of certainty of the referee model regarding the quality of the generated answers.

[0036] S233. Based on the comparison between the confidence score and the preset confidence threshold, decide whether to adopt the generated preference pair.

[0037] Specifically, when the system fails to identify a valid signal through the first two levels, it automatically activates the third-level judge model evaluation mechanism. The system first inputs the user's question containing the complete dialogue context and the large language model's answer into the judge model. This judge model uses the more powerful Qwen3 235B model as its infrastructure, which has stronger reasoning capabilities and knowledge reserves, and can perform in-depth analysis and improvement on the original answer.

[0038] Secondly, the referee model generates candidate correct answers that are superior to the original answers from the large language model based on the input dialogue context. Simultaneously, it outputs a confidence score for the generated results through a built-in self-evaluation module. Specifically, the referee model performs a multi-dimensional quality assessment of its generated candidate correct answers, including verifying the accuracy of the answer, judging the completeness of information, and analyzing logical consistency. Based on the comprehensive evaluation results of these three dimensions, the referee model generates a numerical confidence score between zero and one, which quantifies the degree of certainty the referee model has regarding the quality of the generated answer.

[0039] In real-world applications, for example, when a user asks for a detailed introduction to a museum's history and collections, the large language model provides a relatively brief answer, but the user neither edits the answer nor refactors the query. The system then inputs the question and the original answer into the referee model. The referee model generates a complete answer that includes detailed historical background and information on important collections. Based on content accuracy verification, information coverage analysis, and logical coherence checks, it gives a confidence score of 0.92.

[0040] The system compares the confidence score with a preset confidence threshold of 0.85, which is the optimal value derived from statistical analysis of a large amount of experimental data, ensuring sufficient data volume while maintaining data quality. When the confidence score reaches or exceeds the threshold, the system adopts the generated preference pair; if it falls below the threshold, the data set is discarded. Through this rigorous automated evaluation mechanism, the system can effectively identify and handle marginal cases with unclear implicit feedback signals, ensuring the quality and reliability of the training dataset.

[0041] S3. Generate a structured preference training dataset based on user preference signals. The preference training dataset includes multiple preference pairs, each of which includes a user question, a correct answer, and an incorrect answer.

[0042] Please see Figure 5 This step includes the following steps: S31. Perform unified formatting on user preference signals from different levels to generate data items that conform to the triplet structure; S32. Based on the confidence level of each data item, filter and retain data items with a confidence level higher than a preset threshold; S33. Integrate the filtered data items into a structured dataset for training the direct preference optimization algorithm.

[0043] Specifically, the system first uses a preference pair generation module to uniformly format user preference signals from three levels. Regardless of whether the signal originates from collaborative learning recognition, query reconstruction recognition, or automated evaluation, the system converts it into a standard triplet data structure. Each data item is strictly organized according to a fixed format of user question, correct answer, and incorrect answer, ensuring the uniformity and standardization of the data format. For example, data items from collaborative learning recognition directly use the user-edited text as the correct answer and the model's original answer as the incorrect answer; while data items from query reconstruction recognition retain the incorrect answer field, awaiting further refinement.

[0044] Secondly, the system rigorously filters data items based on their corresponding confidence levels. A comprehensive confidence threshold of 0.85 is set, which is the optimal value derived from extensive experimental analysis of historical training results. For data items identified through first-level collaborative learning, the system assigns them the highest confidence level of 1.0, allowing them to pass the screening directly. For data items from second and third levels, only those with a confidence score of 0.85 or higher are retained for the next stage. This tiered screening mechanism ensures that each preference pair in the final dataset has high quality and reliability.

[0045] Finally, the system batch integrates the qualified data items to construct a structured dataset that can be directly used for training the direct preference optimization algorithm. The dataset is organized according to time order and session identifiers, and each data unit contains complete triple information and corresponding confidence labels. When the accumulated data items reach a preset number, such as 10,000, the system automatically triggers the model fine-tuning process, using this high-quality training data for the optimization iteration of the large language model.

[0046] Through the standardized data processing flow described above, the system can effectively transform user preference signals from different channels and with different confidence levels into training datasets in a unified format, providing reliable data support for continuous model optimization, while ensuring the full automation and high efficiency of the entire training data generation process.

[0047] This invention achieves a fully automated process for training data generation by automatically capturing implicit feedback signals from real user interaction traffic and prioritizing high-confidence signals using a hierarchical judgment mechanism. This effectively overcomes the problems of high cost and poor scalability caused by relying on manual annotation.

[0048] It is understood that those skilled in the art can combine various implementation methods in the above embodiments under the guidance of the above examples to obtain technical solutions with multiple implementation methods.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating training data for a large language model based on traffic replay and implicit feedback, characterized in that, include: Acquire user interaction session data with the large language model in the online service, the interaction session data including user questions, model answers, and user subsequent behaviors; Based on the interactive session data, user preference signals are identified using a hierarchical implicit feedback judgment algorithm; wherein, the hierarchical implicit feedback judgment algorithm prioritizes processing implicit signals with high confidence, including collaborative learning recognition based on user editing behavior, recognition based on user query reconstruction, and automated judgment based on confidence scores; the identification of user preference signals based on the interactive session data using the hierarchical implicit feedback judgment algorithm includes: Based on a preset confidence priority order, the following steps are executed sequentially: collaborative learning recognition based on user editing behavior, recognition based on user query reconstruction, and automated judgment based on confidence score. Specifically, the collaborative learning recognition based on user editing behavior is used to process explicit user editing signals with the highest confidence, the recognition based on user query reconstruction is used to process implicit user feedback signals with medium confidence, and the automated judgment based on confidence score is used to process fuzzy interaction signals with low confidence. The hierarchical implicit feedback judgment algorithm adopts a dynamic progressive processing mechanism, wherein: The collaborative learning recognition based on user editing behavior is processed as the first level. When substantial editing of the model's answer by the user is detected, the highest priority preference pair is generated. The recognition based on user query reconstruction is a second-level process, which is initiated only when no valid signal is recognized in the first level. It identifies user query reconstruction behavior through semantic similarity analysis. The automated judgment based on confidence score is the third level of processing, which is initiated only when no valid signal is identified in the first two levels, and is automatically judged by the referee model. A structured preference training dataset is generated based on the user preference signal. The preference training dataset includes multiple preference pairs, each of which includes a user question, a correct answer, and an incorrect answer.

2. The method for generating training data for a large language model based on traffic replay and implicit feedback according to claim 1, characterized in that, The collaborative learning recognition based on user editing behavior is processed as the first level. When substantial editing of the model's answer by the user is detected, the highest priority preference pair is generated, including: Detect user editing behavior on the model's answer by comparing the model's original answer with the text that the user finally inputs or confirms after the model's answer; The edit distance algorithm is used to calculate the edit distance between the original answer of the model and the edited text by the user, and minor changes are filtered based on a preset edit distance threshold; When the edit distance exceeds the preset edit distance threshold, a preference pair is automatically generated; wherein, the user-edited text is regarded as the correct answer, the original model answer is regarded as the incorrect answer, and the preference pair is assigned the highest confidence level.

3. The method for generating training data for a large language model based on traffic replay and implicit feedback according to claim 2, characterized in that, The method of calculating the edit distance between the original response of the model and the edited text by the user using the edit distance algorithm includes: The Levenstein distance algorithm is used to calculate the minimum number of single-character editing operations between two texts. The system determines whether a user's editing behavior has substantial corrective significance based on a preset edit distance threshold.

4. The method for generating training data for a large language model based on traffic replay and implicit feedback according to claim 1, characterized in that, The user query reconstruction-based identification is a second-level process, initiated only when no valid signal is identified in the first level. It identifies user query reconstruction behavior through semantic similarity analysis, including: Extract two consecutive user questions from an interactive session, and use a text embedding model pre-trained based on contrastive learning to generate semantic vectors for the two user questions. The text embedding model is used to optimize and train for the user query intent recognition task. Calculate the semantic similarity score between two semantic vectors, including performing multi-dimensional feature fusion processing on the semantic vectors; The semantic similarity score is compared with a preset similarity threshold. When the semantic similarity score exceeds the similarity threshold, it is determined that the user has reconstructed the query. When it is determined that the user has restructured the query, the large language model's answer to the previous user's question is marked as an incorrect answer, and a preference pair to be improved is generated, which includes the current user's question, the correct answer to be determined, and the incorrect answer.

5. The method for generating training data for a large language model based on traffic replay and implicit feedback according to claim 4, characterized in that, The multi-dimensional feature fusion processing of the semantic vector includes: The attention mechanism employing query reconstruction dynamically adjusts attention weights based on the time interval and edit distance of user queries; where: The time interval weight is calculated based on the difference in timestamps between two consecutive user questions; the shorter the time interval, the higher the weight. Edit distance weights are calculated based on the user's text editing behavior in the current session; the greater the edit distance, the higher the weight. The time interval weight and edit distance weight are fused into the attention calculation of the semantic vector to generate a weighted semantic similarity score.

6. The method for generating training data for a large language model based on traffic replay and implicit feedback according to claim 1, characterized in that, The automated judgment based on confidence scoring is the third level of processing, which is initiated only when no valid signal is identified in the first two levels. Automated judgment is performed using a referee model, including: The dialogue context, which includes user questions and responses from a large language model, is input into the judge model, which is a high-performance language model. The referee model generates a better answer than the original answer of the large language model as a candidate correct answer, and at the same time outputs a confidence score for the generated result. Based on the comparison between the confidence score and the preset confidence threshold, a decision is made on whether to adopt the generated preference pair.

7. The method for generating training data for a large language model based on traffic replay and implicit feedback according to claim 6, characterized in that, The output includes a confidence score for the generated result, comprising: The referee model performs a multi-dimensional quality assessment on the candidate correct answers it generates, and generates a numerical confidence score based on the results of the multi-dimensional quality assessment. The confidence score is used to quantify the degree of certainty of the referee model regarding the quality of the generated answers.

8. The method for generating training data for a large language model based on traffic replay and implicit feedback according to claim 1, characterized in that, The step of generating a structured preference training dataset based on the user preference signal includes: User preference signals from different levels are uniformly formatted to generate data items that conform to the triplet structure. Based on the confidence level of each data item, data items with a confidence level higher than a preset threshold are selected and retained. The filtered data items are integrated into a structured dataset for training the direct preference optimization algorithm.

Images