Test sample generation method and electronic device

Abstract

The application discloses a test sample generation method and an electronic device. The method comprises the following steps: obtaining a plurality of interaction records of a target system, wherein the interaction records comprise record information generated by the target system in an interaction process and feedback information for a processing result; determining candidate interaction records belonging to a target interaction category according to the feedback information in each interaction record, wherein the target interaction category is used to represent that the processing result of the target system does not meet an expectation; determining candidate information matched with the candidate interaction records from an information source corresponding to the target system; and generating a test sample for testing the target system based on the candidate information.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically to a test sample generation method and electronic device. Background Technology

[0002] With the rapid development of artificial intelligence technology, knowledge-based intelligent question-answering systems have been widely applied in various scenarios such as intelligent customer service, enterprise knowledge management, and academic Q&A. After receiving a user's question, these systems typically need to retrieve relevant information from information sources and generate a corresponding response based on the search results. In this process, the information sources upon which the system relies often contain a large number of documents or data from different sources, with varying timeframes and applicable conditions. Furthermore, the ways users express their input also vary greatly, and these factors can affect the accuracy of the system's response.

[0003] To evaluate and continuously optimize the service quality of the aforementioned systems, it is typically necessary to construct corresponding data samples as test benchmarks. Currently, test samples are mainly constructed manually based on knowledge base content. However, this method is inefficient, costly, and unable to cover the diverse scenarios of large-scale knowledge bases. There are also methods that automatically generate samples using preset templates or rules, but these generate relatively homogeneous samples that are difficult to simulate the complexities of real-world interaction scenarios. Neither of these data sample generation methods meets actual application needs, thus affecting the accuracy of system performance evaluation. Summary of the Invention

[0004] In view of the above, this application provides the following technical solution:

[0005] A method for generating test samples, comprising:

[0006] Acquire multiple interaction records of the target system, the interaction records including the recording information generated by the target system during the interaction process and the feedback information on the processing results;

[0007] Based on the feedback information in each interaction record, candidate interaction records belonging to the target interaction category are determined; wherein, the target interaction category is used to characterize that the processing result of the target system does not meet expectations;

[0008] Candidate information matching the candidate interaction record is determined from the information source corresponding to the target system;

[0009] Based on the candidate information, test samples are generated for testing the target system.

[0010] An electronic device, comprising:

[0011] Processor; and

[0012] Memory for storing the executable instructions of the processor;

[0013] The processor is configured to execute the executable instructions to achieve the following:

[0014] Acquire multiple interaction records of the target system, the interaction records including the recording information generated by the target system during the interaction process and the feedback information on the processing results;

[0015] Based on the feedback information in each interaction record, candidate interaction records belonging to the target interaction category are determined; wherein, the target interaction category is used to characterize that the processing result of the target system does not meet expectations;

[0016] Candidate information matching the candidate interaction record is determined from the information source corresponding to the target system;

[0017] Based on the candidate information, test samples are generated for testing the target system. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a test sample generation method provided in an embodiment of this application;

[0020] Figure 2 A flowchart illustrating another test sample generation method provided in this application embodiment;

[0021] Figure 3 This application provides a schematic diagram of the entire interactive process of a test sample generation method.

[0022] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0024] The terms "first" and "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units may include steps or units not listed, but may include steps or units not listed.

[0025] This application provides a test sample generation method, which can be applied to various scenarios requiring the generation of test samples based on information sources to evaluate system service quality, such as intelligent customer service, enterprise knowledge Q&A, academic Q&A, search engines, and recommendation systems. The entity executing this test sample generation method can be a terminal device, server, cloud computing platform, or other electronic device with data processing capabilities, such as a personal computer, workstation, mobile terminal, virtual machine, or container. The electronic device includes a processor and a memory. The memory stores program instructions. The processor executes the program instructions to obtain recorded information and feedback information generated by the target system during interaction. Based on the feedback information, it filters out candidate interaction records whose processing results do not meet expectations, obtains candidate information matching the candidate interaction records from the information source, and generates test samples for testing the target system based on the candidate information. This application integrates the interaction feedback information generated by the target system in actual operation into the test sample generation process, enabling the generated test samples to accurately reflect the performance defects of the target system in a real interactive environment, thereby improving the relevance and effectiveness of the test samples and providing reliable data support for subsequent system optimization.

[0026] See Figure 1 The diagram illustrates a flowchart of a test sample generation method provided in an embodiment of this application. The method may include the following steps:

[0027] S101. Obtain multiple interaction records of the target system.

[0028] The target system refers to the system to be tested. Its main function is to receive user input information, process it internally, and output the corresponding results. For example, the target system can be a retrieval-augmented generation (RAG) system, a search engine system, an intelligent question-answering system, or a recommendation system.

[0029] Interaction logs refer to the collection of various types of data generated by the target system during a complete interaction. Interaction logs include at least two types of information: recorded information and feedback information. Recorded information refers to the process data generated by the target system at each stage of the interaction, such as the user's input question, candidate documents retrieved by the system, and the system-generated answer. Feedback information refers to the evaluation information returned by the user or evaluator regarding the processing results output by the target system, such as user likes or dislikes for the system's answer, follow-up questions or corrections, and users resubmitting their rephrased questions.

[0030] In one implementation, raw log data is obtained from one or more log storage locations of the target system. The target system typically records data generated by various modules in the form of logs during the processing of interactions. These logs may be stored in different services or devices. Then, based on the interaction identification information (such as session identifiers or request identifiers) contained in the logs, the scattered log data is reassembled into a complete interaction record. The interaction identifier can be used to determine which logs belong to the same interaction, thereby piecing them together into a complete interaction record.

[0031] The interaction records obtained in this step not only include the recorded information generated by the target system during the interaction process, but also the feedback information on the processing results, providing a complete data source for subsequent steps to screen candidate interaction records.

[0032] S102. Based on the feedback information in each interaction record, determine the candidate interaction records that belong to the target interaction category.

[0033] Feedback information is a portion of the acquired interaction records, reflecting the user's or evaluator's assessment of the target system's output. In this embodiment, feedback information is used as a basis for filtering candidate interaction records, allowing inference from the feedback information whether the processing result of the interaction met the user's expectations.

[0034] The target interaction category is a category within the classification results, specifically used to indicate that the target system's processing result did not meet expectations. This failure to meet expectations can be determined through specific signals in the feedback information. For example, when the feedback information includes signals such as user dislikes, user follow-up questions or corrections, user rephrasing of the question, or manual labeling as an error, the processing result of that interaction is determined to have failed to meet expectations. Candidate interaction records refer to interaction records selected from all interaction records that belong to the target interaction category. These candidate interaction records represent instances where the target system performed poorly in actual operation and are the main focus of subsequent analysis.

[0035] In one implementation, for each acquired interaction record, feedback information is extracted. This feedback information can come from various sources, such as user-submitted evaluations; subsequent user actions like follow-up questions, corrections, or resubmitting questions; and automatic system detection signals, such as inconsistencies between the answer and cited evidence, or outdated cited documents. Then, a preset judgment rule is used to determine whether the interaction record belongs to the target interaction category. This judgment rule can be implemented as follows: when a clear negative signal appears in the feedback information, such as a user downvote, a manual labeling as incorrect, or the system detecting an inconsistency between the answer and evidence, the corresponding interaction record is directly identified as a candidate interaction record. When no clear negative signal appears in the feedback information but multiple weak signals exist, such as a user restating the same question and resubmitting it within a short period, a comprehensive judgment is made, and the corresponding interaction record is identified as a candidate interaction record. When the feedback information is insufficient to support the judgment, the interaction record is not identified as a candidate interaction record.

[0036] S103. Determine candidate information that matches the candidate interaction record from the information source corresponding to the target system.

[0037] An information source refers to the data source upon which a target system relies during its operation, and its specific form depends on the type of the target system. For example, in a scenario where the target system is a Retrieval Enhanced Generation (RAG) system, the information source is a knowledge base, which stores knowledge units such as documents, rules, and clauses used to answer user questions. In a scenario where the target system is a search engine, the information source is a web page index, which stores web page data after crawling and indexing. In a scenario where the target system is a recommendation system, the information source is an item database, which stores information about recommendable items and their corresponding tags or attributes. In a scenario where the target system is an automated decision-making system, the information source is a rule base, which stores rules, conditions, and exception clauses used for decision-making. The core function of an information source is to provide the target system with the basic data upon which to generate output results, and its form can be a database, index, knowledge graph, file system, or other structured or unstructured data storage format.

[0038] Candidate information refers to information units obtained from the information source that match candidate interaction records. The specific content of candidate information depends on the interaction pattern reflected by the candidate interaction records. For example, if candidate interaction records show that the target system frequently makes mistakes when answering a certain type of question, then the candidate information could be new version rules and old version similar rules related to that type of question.

[0039] In one implementation, candidate interaction records can be analyzed to extract information related to the interaction process and results. For example, candidate interaction records may contain user-inputted questions, candidate documents retrieved by the target system, and the final answer generated by the target system. Based on the extracted information, it is determined what information needs to be obtained from the information source to generate test samples. For instance, it is possible to statistically analyze which combinations of information frequently appear in the candidate interaction records, identify these frequent combinations as target combinations, and then obtain information matching these target combinations from the information source as candidate information. In a scenario where the target system is a retrieval-enhanced generation system, if the candidate interaction records frequently contain the combination of user asking questions using relative time, the system retrieving older documents, and the system using older rules to generate answers, then the new and old rules, as well as the correlation information between them, are obtained from the knowledge base as candidate information.

[0040] In this way, candidate information matching the candidate interaction records is obtained from the information source. The obtained candidate information is related to the actual weak links of the target system, providing content material for the subsequent generation of test samples.

[0041] S104. Based on the candidate information, generate test samples for testing the target system.

[0042] A test sample refers to a set of data used to test a target system, including input information and corresponding standard result information. After the test sample is input into the target system, the target system produces an output result. By comparing this output result with the standard result information in the test sample, the performance of the target system on that sample can be evaluated.

[0043] In one implementation, the content of the test sample is determined based on candidate information, which includes information units matching candidate interaction records, such as correct evidence and distracting evidence. Based on these information units, a question simulating a real user interaction (i.e., input information) and the corresponding correct answer (i.e., standard result information) can be generated. Then, the generated input information and standard result information are combined to form the test sample. This test sample can be used to test the target system. For example, the input information is input into the target system, the actual output result of the target system is obtained, and the actual output result is compared with the standard result information to evaluate the performance of the target system on the sample. When generating the test sample, the expression style of the input information can also be determined based on the expression characteristics in the candidate interaction records, making the generated input information linguistically close to the questioning habits of real users.

[0044] This generates test samples for testing the target system. These test samples are associated with the weak points of the target system in actual interaction and can accurately reflect the performance defects of the target system in a real interactive environment.

[0045] This application provides a test sample generation method. This method acquires interaction records containing feedback information generated by a target system during actual interaction, filters out candidate interaction records whose processing results do not meet expectations based on this feedback information, obtains candidate information matching the candidate interaction records from the information source corresponding to the target system, and finally generates test samples based on this candidate information. This method integrates the interaction feedback of the target system in a real operating environment into the test sample generation process, making the generated test samples relevant to the actual weaknesses of the target system. This improves the relevance and effectiveness of the test samples, overcoming the technical defect of test sample generation methods in conventional processing modes where samples are disconnected from real application scenarios. It provides a reliable data foundation for accurately evaluating the service quality of the target system and precisely locating the direction of system optimization.

[0046] In some embodiments of this application, the process of obtaining multiple interaction records of a target system may include the following steps:

[0047] S201. Obtain the recorded information generated by the target system at multiple stages during the interaction process, as well as the feedback information on the processing results.

[0048] In processing a complete interaction, a target system typically undergoes multiple sequential processing steps, each of which can be called a stage. Stages are arranged chronologically according to the interaction process, meaning there is a sequential relationship between them; the output of one stage may serve as the input for the next. The division of stages is related to the specific processing flow of the target system; different target systems may have different stage division methods, such as primarily dividing them into input stages, processing stages, and output stages.

[0049] For example, in a scenario where the target system is a retrieval-enhanced generation system, the interaction process can be specifically divided into the input stage, context stage, data source stage, retrieval stage, ranking stage, evidence citation stage, output stage, and feedback stage. Specifically, the input stage generates records including the original user input received by the target system; the context stage generates records including related interaction information prior to the current interaction, such as historical question-and-answer records in multi-turn dialogues; the data source stage generates records including structural features of the data sources accessed by the target system, such as document titles, table of contents, chapter divisions, version numbers, effective dates, and scope of application; the retrieval stage generates records including the results and matching information of the candidate documents or document fragments retrieved from the knowledge base, such as the list of retrieved candidate documents and the retrieval scores of each document; the ranking stage generates records including the results of the target system retrieving candidate documents or document fragments from the knowledge base, such as the list of retrieved candidate documents and the retrieval scores of each document; the ranking stage generates records including the results of the target system retrieving candidate documents or document fragments from the knowledge base, such as the retrieval scores of each document; the evidence citation stage generates records including the results of the target system retrieving candidate documents or document fragments ... results of the target system retrieving candidate documents, such as the The information recorded in the sequencing phase includes the order information after the target system sorts the recalled candidate data, such as the reordered document ranking results; the information recorded in the evidence citation phase includes the data source information on which the target system generates the output results, such as the specific document fragments or evidence cited when generating the answer; the information recorded in the output phase includes the output content information generated by the target system for the current interaction, such as the final generated answer text; and the information recorded in the feedback phase includes the evaluation feedback information received by the target system regarding the output content, such as user likes, dislikes, follow-up questions and corrections, and users resubmitting after rephrasing their questions. These phases together constitute the full-link phase division of the target system in processing a complete interaction.

[0050] In one implementation, based on the processing flow of the target system, the stages included in the interaction process of the system and the start and end boundaries of each stage are determined in advance, and log recording points or data collection points are set at the processing nodes corresponding to each stage. When the target system processes the interaction, the data generated in each stage is automatically recorded.

[0051] In scenarios where the target system is a retrieval enhancement generation system, interaction logs are collected from online or offline services. Log sources include user questions (i.e., the original questions entered by the user), session context (i.e., historical information from multi-turn dialogues), retrieval candidate documents and document fragments, retrieval scores, reordered results, generated answers, answer citation evidence, user feedback, and records of subsequent user follow-up questions or revisions. In scenarios with human evaluation or standard answers, reference answers, standard evidence, and human error type annotations may also be included. This recorded information and feedback information are typically stored as logs on different services or devices within the target system and retrieved through a unified collection interface.

[0052] The scattered logs collected using the above methods are concatenated into a complete interaction chain, forming a standard record, which is represented as follows:

[0053] x i = {q i , c i D i , R i , K i , a i , e i , f i , t i}

[0054] Where, x i q represents the complete interaction record corresponding to the i-th interaction; i Indicates a user problem; c i Indicates the session context; D i Represents a set of candidate documents; R i Indicates the recall results and search score; K i Indicates the rearrangement result; a i Indicates the generation of the answer; e i Indicates citing evidence or excerpts of evidence; f i This indicates user feedback or evaluation feedback; t i These fields represent time and document version information. Together, they form the data structure of a complete interaction record.

[0055] S202. Associate the recorded information and feedback information belonging to the same interaction process to obtain the interaction record.

[0056] When the target system processes an interaction, the recorded and feedback information generated at each stage is stored separately as independent log entries. To obtain a complete interaction record, this scattered data needs to be reorganized according to the interaction process to which it belongs.

[0057] In one implementation, interaction identifier information is extracted from each acquired record and feedback message. This interaction identifier is a unique identifier generated by the target system for each interaction when logging, such as a session identifier or request identifier. This identifier accompanies the entire interaction process, from user input to system output and feedback reception, and is carried in all log records related to this interaction. Then, records and feedback messages carrying the same interaction identifier are merged, splicing the scattered log data into a complete interaction chain, combining them into a complete interaction record. This interaction record is the standard record mentioned above, containing complete information from user input to system processing and then to user feedback.

[0058] In this way, the recorded and feedback information scattered across various stages of the target system is linked into a complete interaction record. This interaction record contains complete data from the entire interaction process of the target system, with recorded information covering the processing at each stage and feedback information covering the user's evaluation of the processing results, providing a complete data foundation for subsequent screening of candidate interaction records.

[0059] For example, consider a retrieval-enhanced generation system as the target system. In a user question-and-answer interaction, the system records the user's question "Have there been any changes to this year's car insurance strategy?" during the input phase, the historical context of the user's session during the context phase, the list of candidate documents retrieved from the knowledge base and the retrieval scores of each document during the retrieval phase, the rearranged document order during the sorting phase, the specific document fragments cited when generating the answer during the evidence citation phase, the generated answer "According to the V2.0 strategy, premiums have decreased by an average of 15%" during the output phase, and the user's dislike of the answer and subsequent follow-up question "Is the strategy you mentioned still effective?" during the feedback phase. This data is scattered across different log stores within the system, linked by the same request identifier, and pieced together into a complete interaction record. This interaction record simultaneously contains recorded information (user question, historical context, retrieved documents and scores, rearranged order, cited evidence, generated answer) and feedback information (dislikes, follow-up questions), providing a complete data foundation containing end-to-end information for subsequent screening of candidate interaction records.

[0060] By acquiring the recorded information generated by the target system at multiple stages during the interaction process, as well as the feedback information on the processing results, and by associating the recorded information and feedback information belonging to the same interaction process, the log data, originally scattered across different stages and storage locations, is reorganized into a complete interaction record containing end-to-end information. This interaction record fully reflects the target system's processing procedure in a single interaction and the user's evaluation of the processing results, solving the problem of difficulty in correlation caused by data dispersion, and providing a complete and traceable data foundation for subsequent analysis based on the interaction record.

[0061] In some embodiments of this application, the process of determining candidate information matching candidate interaction records from the information source corresponding to the target system may include the following steps:

[0062] S301. Extract multiple interaction path information from candidate interaction records.

[0063] Candidate interaction records are those where the target system's processing result did not meet expectations. For each candidate interaction record, one or more interaction path information are extracted according to preset path extraction rules. Interaction path information refers to the combination of information extracted from candidate interaction records that characterizes the features of the interaction. In the scenario of a retrieval enhancement generation system, a complete question-and-answer interaction can be broken down into multiple processing stages, including the user question input stage, the conversation context processing stage, the document retrieval stage, the retrieval result reordering stage, the evidence citation stage, the answer generation stage, and the user feedback stage. Each stage can be encoded as an event, and each event includes the stage identifier to which it belongs, the specific type of the event, and the positive or negative attributes of the event. Positive attributes indicate that the processing result meets expectations, while negative attributes indicate that the processing result does not meet expectations. After the above event-based encoding, each interaction record is encoded as an event sequence.

[0064] For example, an interaction that fails to meet expectations can be encoded as a sequence of events {e}. q e c e d e r e k e v e a e f}. Among them, e q The event representing the user issue phase, e c Events representing the session context phase, e d Indicates events in the document phase, e r Indicates an event during the retrieval phase, e k e represents the events during the rearrangement phase. v The event in the evidence citation stage, e a e represents an event during the answer generation phase. f This refers to events occurring during the user feedback phase.

[0065] When extracting interaction path information from event sequences, various extraction methods can be employed. Candidate paths include continuous paths, non-continuous combination paths, and path templates with order constraints. A continuous path is a chain of events that occurs consecutively according to the processing flow of the retrieval enhancement generation system; a non-continuous combination path is a combination of events that are not necessarily adjacent but jointly lead to failure, such as term aliases, low ranking of standard evidence, or missing answers; a path template with order constraints requires several key events to appear in a specified order, but allows other events to exist in between. Through these various extraction methods, multiple different types of interaction path information can be extracted from candidate interaction records.

[0066] S302. Count the frequency of each interaction path information in each candidate interaction record.

[0067] Count the frequency of each interaction path information in all candidate interaction records. The frequency of occurrence represents the number of candidate interaction records containing that interaction path information.

[0068] Specifically, since each candidate interaction record corresponds to an independent interaction process, one or more interaction path information can be extracted from each candidate interaction record. Identical interaction path information is matched and counted across multiple candidate interaction records. For example, if interaction path information A is extracted from all 30 candidate interaction records, then the frequency of interaction path information A is 30; if interaction path information B is extracted from only 2 candidate interaction records, then the frequency of interaction path information B is 2. Through this method, the frequency of each interaction path information in the candidate interaction records can be obtained.

[0069] S303. The interaction path information whose frequency meets the frequency condition is determined as the target path information.

[0070] Based on the frequency of occurrence of each interaction path information obtained from statistics, the interaction path information whose frequency meets the frequency condition is determined as the target path information. The specific implementation method of the frequency condition will be further described in subsequent embodiments.

[0071] S304. Obtain information from the information source that matches the target path information as candidate information.

[0072] Based on the target path information, information matching the target path information is retrieved from the information source corresponding to the target system as candidate information. The specific form of the information source depends on the type of the target system. For example, in the scenario where the target system is a retrieval-enhanced generation system, the information source is a knowledge base, which stores knowledge units such as documents, rules, and clauses used to answer user questions.

[0073] In the context of a retrieval enhancement generation system, before acquiring candidate information, the document knowledge base can be parsed to construct a document knowledge graph. Specifically, the document knowledge base is parsed, retaining information such as document title, table of contents level, chapter, paragraph, segment, table, version number, effective date, and scope of application. Then, knowledge units such as entities, rules, conditions, indicators, actions, exceptions, and time are extracted to form a knowledge graph G=(V,E). The nodes of the knowledge graph can include document nodes, chapter nodes, paragraph nodes, segment nodes, entity nodes, rule nodes, condition nodes, time and version nodes, table cell nodes, etc.; the edges of the knowledge graph can include containment relationships, reference relationships, alias relationships, equivalence relationships, update relationships, validity period relationships, application relationships, exception relationships, similarity relationships, and evidence relationships, etc.

[0074] After constructing the knowledge graph, candidate information is retrieved from it based on the target path information. Specifically, the target path information can first be converted into knowledge graph sampling constraints, and then evidence subgraphs can be extracted from the knowledge graph based on these constraints. Sampling constraints are used to limit the types of nodes, edges, correct evidence, distracting evidence, user expression styles, and expected error types that should appear in the graph. Evidence subgraphs typically include correct evidence supporting the correct answer, related documents, necessary conditions, time or version nodes, and one or more distracting pieces of evidence that are easily misrecalled or miscited. Distracting evidence is used to simulate content similar to but not suitable as the basis for an answer in a real retrieval process. The final candidate information obtained includes both correct and distracting evidence extracted from the knowledge graph.

[0075] In this embodiment, interaction path information is extracted from candidate interaction records, and the frequency of each interaction path information in the candidate interaction records is statistically analyzed. Interaction path information whose frequency meets the frequency criteria is determined as target path information. Information matching the target path information is then obtained from the information source as candidate information, ensuring that the obtained candidate information corresponds to frequently occurring interaction patterns in the candidate interaction records. Specifically, the extraction of interaction path information is based on breaking down the interaction record into a multi-stage event sequence and performing event-based encoding, enabling the extracted path information to accurately represent key features in the interaction process. The determination of target path information is based on frequency statistics and filtering of path information extracted from multiple candidate interaction records, ensuring that the target path information reflects recurring problem patterns in the actual operation of the target system. The acquisition of candidate information is based on targeted matching of target path information from the information source, ensuring that the acquired candidate information precisely corresponds to the target path information. This method achieves a complete transformation from candidate interaction records to candidate information, avoiding the disconnect between samples and actual problems caused by blindly extracting information from the information source, and providing accurate content material for subsequent generation of test samples.

[0076] Furthermore, in the embodiments of this application, the process of determining the interaction path that appears frequently enough to meet the frequency condition as the target path information may include method one and method two.

[0077] Method 1: Interaction path information with an occurrence frequency of not less than a first threshold is identified as candidate path information, and the candidate path information is identified as target path information.

[0078] After calculating the frequency of occurrence for each interaction path, a preliminary screening is performed based on these frequency statistics. Specifically, a first threshold is pre-set, and interaction paths with a frequency not less than (i.e., greater than or equal to) the first threshold are identified as candidate paths. The specific value of the first threshold can be determined based on the actual application scenario and the total number of candidate interaction records. For example, it can be set to a fixed value (e.g., a frequency of occurrence not less than 5), or it can be determined by multiplying the total number of candidate interaction records by a proportion (e.g., a frequency exceeding 10% of the total number of candidate interaction records), or other methods can be used to determine it.

[0079] In this approach, the target path information is the same as the candidate path information. That is, using the aforementioned filtering based on a first threshold as the sole filtering condition, interaction path information with a frequency not less than the first threshold is directly identified as the target path information. This method only needs to count the frequency of occurrence in candidate interaction records, without needing to obtain statistical data from other interaction records, making it relatively simple and enabling rapid filtering of target path information.

[0080] Method 2: Interaction path information with an occurrence frequency of not less than a first threshold is identified as candidate path information. The occurrence frequency of candidate path information in the first interaction record is obtained, and interaction path information among the candidate path information whose occurrence frequency in the first interaction record is not greater than a second threshold is identified as target path information.

[0081] The first interaction record is the interaction record other than the candidate interaction records among multiple interaction records. Specifically, the frequency of occurrence of candidate path information in the first interaction record is obtained. This frequency of occurrence represents the number of interaction records in the first interaction record that contain the candidate path information.

[0082] After obtaining the frequency of occurrence of candidate path information in the first interaction record, a second threshold is used as a filtering condition to further filter the candidate path information. The second threshold is used to filter out interaction path information with a lower frequency of occurrence in the first interaction record. Interaction path information whose frequency of occurrence in the first interaction record is not greater than the second threshold is determined as target path information.

[0083] In Method Two, the selected target path information simultaneously meets two conditions: its frequency of occurrence in candidate interaction records is not less than a first threshold (i.e., it frequently appears in interactions where the processing result does not meet expectations), and its frequency of occurrence in the first interaction record is not greater than a second threshold (i.e., it appears less frequently in interactions where the processing result meets expectations). Such target path information represents the interaction patterns in the target system that frequently occur in interactions where the processing result does not meet expectations but less frequently occur in interactions where the processing result meets expectations—in other words, the interaction features that best distinguish the quality of the target system's processing results. In Method Two, the difference between the frequency of path information in candidate interaction records and its frequency in the first interaction record quantifies the path information's ability to distinguish the quality of the target system's processing results. The greater this difference, the more accurately the path information reflects the weak points of the target system.

[0084] For example, consider a retrieval enhancement generation system as the target system. Assume that multiple interaction path information is extracted and statistically analyzed from candidate interaction records (i.e., interaction records where the processing result did not meet expectations). The first interaction path information represents the path of relative time expression, multiple version documents, older versions being recalled earlier, and expired answers, with a frequency of 30. The second interaction path information represents the path of colloquial abbreviations, lower retrieval recall, and missing answers, with a frequency of 15. The third interaction path information represents the interaction path of missing conditional constraints, ignoring exception clauses, and answers violating constraints, with a frequency of 8. Setting a first threshold of 10, the first and second interaction path information with a frequency of at least 10 are identified as candidate path information. The third interaction path information with a frequency of 8 is not included in the candidate path information.

[0085] If Method 1 is used, the first and second interaction path information mentioned above are determined as the target path information. If Method 2 is used, the frequency of occurrence of the candidate path information in the first interaction record (i.e., the interaction record where the processing result meets expectations) needs to be obtained. For example, if the frequency of occurrence of the first interaction path information in the first interaction record is 2, and the frequency of occurrence of the second interaction path information in the first interaction record is 12, if the second threshold is 5, then the frequency of occurrence of the first interaction path information in the first interaction record (i.e., 2) is not greater than 5, and it is retained as the target path information; the frequency of occurrence of the latter (i.e., 12) in the first interaction record is greater than 5, and it is eliminated. Finally, the first interaction path information is determined as the target path information.

[0086] Of the two methods described above, Method 1 uses a first threshold to filter frequently occurring interaction path information from candidate interaction records, achieving simplicity and efficiency, and is suitable for scenarios requiring rapid acquisition of target path information. Method 2, building upon Method 1, further introduces a first interaction record as a comparison reference, using a second threshold to eliminate interaction path information that also frequently appears in successful interactions. This makes the filtered target path information more specific and more accurately reflects the weaknesses of the target system in actual operation. In practical applications, Method 1 or Method 2 can be flexibly selected according to specific needs, or a combination of both methods can be used.

[0087] In this embodiment of the application, the interaction path information is a combination of record information corresponding to multiple stages in the candidate interaction record, and the interaction path information includes at least one of the following types:

[0088] The first type consists of a combination of recorded information corresponding to at least two consecutively arranged stages. For example, in the scenario of a retrieval-enhanced generation system, a complete question-and-answer interaction can be broken down into multiple processing stages. These stages are arranged sequentially according to the time order in which the target system processes the interaction, including the user question input stage, the session context processing stage, the document retrieval stage, the retrieval result reordering stage, the evidence citation stage, the answer generation stage, and the user feedback stage. Each stage can be encoded as an event, and each event includes the stage identifier to which it belongs, the specific type of the event, and the positive or negative attributes of the event. The positive attributes indicate that the expectation is met, and the negative attributes indicate that the expectation is not met.

[0089] The interaction path information corresponding to the first type mentioned above refers to a chain of events that occur consecutively in the aforementioned chronological order. For example, in an interaction, events occur sequentially such as the user asking a question using relative time expression, the system retrieving multiple versions of the document from the knowledge base, the older version of the document ranking higher in the search results and being used to generate an answer, and the user reporting that the answer has expired. These events are arranged consecutively in chronological order and together constitute an interaction path information formed by the record information corresponding to the consecutively arranged stages, such as the interaction path information of relative time expression, multiple versions of the document, older version being recalled first, and answer expiring.

[0090] This reflects the cascading transmission of errors through the target system's processing chain. In a retrieval-enhanced generation system, the way a question is phrased during user input affects the recall results during retrieval, which in turn affects the document order during ranking, which in turn affects the evidence selection during evidence citation, and finally, the evidence selection during evidence citation ultimately impacts the output quality during answer generation and the evaluation during user feedback. The interaction path information represented by the first type of representation can fully capture this causal transmission chain.

[0091] The second type consists of a combination of recorded information corresponding to at least two non-contiguous stages. That is, the second type refers to a combination of recorded information that is not necessarily temporally adjacent but appears together in the same interaction. For example, the events "user asks a question using a term alias" and "standard evidence ranks low in the search results" may not be adjacent (there may be other events in between), but they appear together in the same interaction, jointly causing the processing result to fail to meet expectations. This constitutes a non-contiguous combination: "term alias, low ranking of standard evidence, and missing answer," used to represent the interaction path information corresponding to the second type.

[0092] The second type of interaction path information can capture cross-stage collaborative failure modes. For example, in a retrieval-enhanced generation system, some failure modes are not caused by errors in a single stage, but by the combined effects of different problems in multiple stages. These events are not necessarily sequential in time, but their combined occurrence in the same interaction has specific failure characterization significance, which is precisely designed to capture such cross-stage collaborative failure modes.

[0093] The third type consists of recorded information from at least two stages arranged in a specified chronological order. In other words, this type requires several key events to occur in a specified sequence, but allows other events to occur in between, and does not require these events to be consecutive in time. For example, when analyzing the performance of a target system in a multi-turn dialogue, the event "user omits key conditions" can be required to occur before the event "system ignores exception clauses," but other events can occur in between (such as the system retrieving documents, generating intermediate answers, etc.). The interaction path information corresponding to this combination can also be called a sequence constraint combination.

[0094] It can capture combinations of events that are causally ordered but not adjacent. In some interaction scenarios, the occurrence of a previous event creates the conditions for the occurrence of a subsequent event, but there may be other processing steps between them. For example, if a user omits a key condition, the system may fail to find a document containing that condition during retrieval, leading the system to ignore exception clauses and generate an answer that violates the constraints. There is a causal relationship between these two events, but they are not consecutive in time, such as being separated by multiple steps like retrieval, sorting, and generation. The interaction path information corresponding to the third type is precisely designed to capture such combinations of events that are causally ordered but not adjacent.

[0095] When extracting interaction path information from candidate interaction records, one or more of the three types mentioned above can be selected for extraction based on the analysis objectives and scenario requirements. Specifically, for each candidate interaction record, a continuously arranged event chain can be extracted as the interaction path information corresponding to the first type. Multiple events spanning different stages can be extracted as the interaction path information corresponding to the second type. Events satisfying a specified sequential relationship can be extracted as the interaction path information corresponding to the third type. Combinations of multiple types can be extracted simultaneously to cover different types of failure modes.

[0096] By extracting various types of interaction path information, we can obtain different combinations from candidate interaction records. These combinations cover multiple failure modes, including continuous propagation, cross-stage coordination, and causal sequence, providing a diverse analytical foundation for subsequent frequency statistics and path selection. Different types of interaction path information can be used individually or in combination as needed in practical applications to adapt to different failure mode analysis requirements.

[0097] In some embodiments of this application, the process of determining candidate information matching the target path information from the information source corresponding to the target system may include the following steps:

[0098] S401. Generate sampling constraints based on the target path information.

[0099] Target path information refers to the interaction paths that appear in candidate interaction records with a frequency that meets a frequency condition. It represents the interaction patterns corresponding to the recurring weaknesses in the target system during actual operation. In order to obtain information from the information source that can be used to generate test samples, the target path information needs to be converted into executable sampling constraints.

[0100] Sampling constraints include at least one of the following:

[0101] The type constraints of the information units involved in the target path information. For example, if the target path information involves version documents, the constraints can limit the knowledge units that need to be obtained to document type; if the target path information involves business rules, the constraints can limit the knowledge units that need to be obtained to rule type.

[0102] Constraints on the types of relationships between information units. For example, if the target path information involves a comparison between old and new versions, the constraint can limit the acquisition of documents with an "updated" relationship; if the target path information involves interference from similar clauses, the constraint can limit the acquisition of rules with a "similar" relationship.

[0103] The constraint on the extraction of correct evidence used to generate correct samples. Correct evidence refers to the knowledge units that support the correct answer. This constraint is used to limit which information units are extracted from the information source as the basis for generating the standard answer.

[0104] The constraint on the extraction of distracting evidence is used to define which information units are extracted from the information source as distractors. Distracting evidence refers to knowledge units that are similar to correct evidence but are incorrect or not applicable to the current problem. This constraint limits which information units are extracted from the information source as distractors to simulate false recall in the real retrieval process.

[0105] The specific content of sampling constraints varies depending on the target path information. For example, if the target path information includes relative time expression, multiple versions of documents, older versions being recalled earlier, and expired answers, the sampling constraints may include: needing to obtain knowledge units of document type (information unit type constraint), needing to obtain knowledge units with "update" and "similar" relationships (association relationship type constraint), needing to obtain rules of the currently valid version as correct evidence (correct evidence extraction condition constraint), and needing to obtain obsolete old version rules as interfering evidence (interfering evidence extraction condition constraint).

[0106] S402. Based on the sampling constraints, at least one information unit that satisfies the sampling constraints is obtained from the information source as candidate information.

[0107] After generating the sampling constraints, the information source is searched or queried according to the sampling constraints to obtain at least one information unit that satisfies the sampling constraints as candidate information.

[0108] In scenarios where the target system is a retrieval-enhanced generation system, the knowledge base can be pre-parsed to construct a knowledge graph. The knowledge graph consists of nodes and edges. Nodes represent knowledge units in the knowledge base (such as documents, rules, conditions, versions, etc.), while edges represent the relationships between knowledge units (such as update relationships, similarity relationships, reference relationships, etc.). After constructing the knowledge graph, sampling constraints can be converted into graph query conditions. By querying the knowledge graph, nodes and edges that satisfy the constraints can be obtained as candidate information.

[0109] Using the above method, sampling constraints were generated based on the target path information, and at least one information unit satisfying the constraints was obtained from the information source as candidate information. The obtained candidate information corresponds precisely to the target path information, containing both correct evidence needed to generate correct samples and interfering evidence used to simulate false recall scenarios. In this way, the candidate information provides precise content material for the subsequent generation of test samples; correct evidence is used to generate standard answers, and interfering evidence is used to construct deceptive test scenarios, thereby enabling the generated test samples to effectively expose the performance defects of the target system under the interaction mode corresponding to the target path information.

[0110] See Figure 2 The document illustrates a flowchart of another test sample generation method provided in this application embodiment. This method includes a process for verifying the generated test samples. Specifically, the process may include the following steps:

[0111] S501. Obtain multiple interaction records of the target system.

[0112] S502. Based on the feedback information in each interaction record, determine the candidate interaction records that belong to the target interaction category.

[0113] S503. Determine the candidate information that matches the candidate interaction record from the information source corresponding to the target system.

[0114] The specific implementation process of steps S501 to S503 has been described in the foregoing embodiments and will not be detailed here.

[0115] S504. Based on the candidate information and the target expression style information, generate the input information and standard results for the candidate test samples.

[0116] Candidate information matching the target path information was obtained from the information source. This candidate information included both correct and interfering evidence. When generating test samples, the content of the test samples, namely the input information and the standard result information, first needs to be determined.

[0117] Input information refers to questions simulating real users posing to the target system. Input information generation is based on candidate information and target expression style information. Candidate information provides the knowledge content related to the input information, such as new and old rules, which determines the theme the input information should revolve around. Target expression style information refers to user expression feature constraints extracted from candidate interaction records, used to guide the generative model to generate input information that closely resembles the questioning habits of real users in terms of language expression. Target expression style information can include features such as commonly used question phrases, terminology usage habits, and language style.

[0118] In one implementation, the knowledge topics and content scope of the input information can be determined based on the correct and contradictory evidence contained in the candidate information. Then, based on the target expression style information, the language expression of the input information is determined, generating input information that conforms to the questioning habits of real users. The standard result information refers to the correct answer corresponding to the input information, that is, the correct result that the target system should ideally output for this input information. The standard result information is generated based on the correct evidence in the candidate information, ensuring that the standard result information is based on verifiable knowledge and is accurate.

[0119] For example, regarding the target path information of "relative time expression, multiple version documents, older versions prioritized, and expired answers," the candidate information includes the original text of the rule in the currently valid version (i.e., correct evidence) and the original text of similar rules in the obsolete version (i.e., interfering evidence). The target expression style information is to use relative time expression. The input information generated based on the candidate information and the target expression style information is "Are car insurance premiums cheaper this year?", which uses the relative time expression "this year." The standard result information generated based on the correct evidence is "According to the V3.0 rule, car insurance premiums are linked to the number of claims; a 30% discount is available for no claims, and a 25% increase is available for two claims."

[0120] S505. Based on the target path information, generate path identification information for candidate test samples.

[0121] Path identification information corresponds to target path information and is used to characterize the unmet expected interaction patterns corresponding to candidate test samples. Path identification information is a labeled representation of target path information in test samples; it transforms abstract target path information into a recognizable field within the test sample.

[0122] Path identification information can include event type identifiers for each stage corresponding to the target path information. For example, for the target path information "relative time expression, multiple version documents, older version recall prioritized, answer expired", the corresponding path identification information can be represented as a sequence of event type identifiers, such as ["relative time expression", "multiple versions coexist", "older version recall prioritized", "referencing expired version", "answer expired"]. Path identification information can also take other forms that can uniquely identify the interaction mode.

[0123] When a test sample is used to test a target system, if the target system performs poorly on the sample (i.e., the output results are inconsistent with the standard results), the corresponding target path information can be located directly based on the path identifier information carried by the sample. This allows for a quick determination of the type of interaction mode that the sample targets that did not meet expectations, without the need to re-analyze the failed path.

[0124] S506. Verify the candidate test samples.

[0125] After generating candidate test samples, these samples need to be validated to verify whether their quality meets the requirements. Validation may include at least one of the following:

[0126] First, verify whether the standard result information in the candidate test samples can be derived from the correct evidence information in the candidate information. This verification ensures that the standard result information has a traceable knowledge basis, avoiding errors or inaccuracies in the standard result information.

[0127] Second, verify whether the result information in the candidate test samples incorrectly depends on interfering evidence information in the candidate information. This verification ensures that the test sample design is as expected, i.e., the standard result information is generated based on correct evidence rather than interfering evidence.

[0128] Third, verify whether the candidate test samples meet the path characteristics corresponding to the target path information. This verification ensures that the interaction characteristics of the candidate test samples match the interaction pattern represented by the target path information, that is, the sample can indeed trigger the failure mechanism corresponding to the target path information.

[0129] Fourth, verify whether the input information in the candidate test samples conforms to the expression style represented by the target expression style information. This verification ensures that the input information is close to the questioning habits of real users in terms of language expression, avoiding the generated input information being too templated or formal.

[0130] All of the above verification items can be executed, or one or more of them can be selected to be executed according to actual needs.

[0131] S507. Use the candidate test samples that have passed the verification as test samples.

[0132] After verifying the candidate test samples, the candidate test samples that pass the verification will be output as the final test samples. Candidate test samples that fail the verification can be discarded, or they can be regenerated or modified and verified again.

[0133] By validating candidate test samples, including verifying whether the standard result information can be derived from correct evidence, whether the result information is incorrectly dependent on interfering evidence, whether the candidate test samples meet the path characteristics corresponding to the target path information, and whether the input information conforms to the expression style represented by the target expression style information, the quality of the test samples is guaranteed from multiple dimensions. This ensures that the final output test samples have both knowledge basis and clear test objectives, and that the input information is close to the questioning habits of real users in terms of language expression, effectively avoiding the problem of test samples not matching real application scenarios.

[0134] In this embodiment, the target system is a Retrieval-Augmented Generation (RAG) system. A RAG system is a processing system that first retrieves and then generates an answer. Its processing flow involves retrieving the most relevant document fragments from a knowledge base to the user's input question, inputting the retrieved document fragments and the user question into a large language model, and then generating an answer to the user's question from the large language model. The RAG system can be applied to scenarios such as intelligent customer service, enterprise knowledge Q&A, and academic Q&A.

[0135] In the context of a retrieval enhancement generation system, interaction logs are the log information generated by the system at multiple stages during a single question-and-answer interaction. Specifically, the question-and-answer interaction process of a retrieval enhancement generation system can be broken down into several stages: user question input, session context processing, document retrieval, search result reordering, evidence citation, answer generation, and user feedback. Each stage generates corresponding log information. The interaction logs contain complete end-to-end information from user question input to system answer generation and receiving user feedback.

[0136] In the context of a retrieval enhancement generation system, the information source is a knowledge base. The knowledge base stores the original knowledge materials upon which the retrieval enhancement generation system relies when generating answers, including but not limited to knowledge units such as documents, document fragments, rules, clauses, and tables.

[0137] When the target system is a retrieval-enhanced generation system, the process of determining candidate information that matches candidate interaction records from the information source corresponding to the target system may include the following steps:

[0138] S601. Determine sampling constraints based on candidate interaction records.

[0139] In the context of a retrieval-enhanced generation system, candidate interaction records are those where the processing results did not meet expectations. Analyzing these candidate interaction records allows us to determine sampling constraints. These constraints limit which knowledge units are retrieved from the knowledge base to generate test samples.

[0140] Sampling constraints can be determined based on interaction path information in candidate interaction records. Interaction path information characterizes frequently occurring interaction patterns in candidate interaction records. For example, interaction path information such as relative time expression, multiple version documents, older versions being recalled first, and expired answers indicate that when a user asks a question using relative time expression, the system is likely to retrieve obsolete older versions of documents and use them to generate answers, leading to outdated answers. Sampling constraints may include: requiring the acquisition of document-type knowledge units, requiring the acquisition of knowledge units with update and similarity relationships, requiring the acquisition of rule content from the currently valid version as correct evidence, and requiring the acquisition of similar rule content from obsolete versions as interfering evidence. The specific types of sampling constraints have been described in detail in the foregoing embodiments and will not be repeated here.

[0141] S602. Obtain the knowledge graph corresponding to the knowledge base.

[0142] A knowledge graph is a structured representation of a knowledge base, used to characterize knowledge units and their relationships within the knowledge base. In the context of retrieval-enhanced generation systems, the knowledge base stores a large amount of knowledge material such as documents, rules, and clauses. These knowledge materials have complex relationships, such as update relationships between documents, similarity relationships between rules, and reference relationships between clauses. A knowledge graph provides a structured representation of these knowledge units and their relationships through nodes and edges.

[0143] When constructing a knowledge graph, the knowledge base is first parsed to extract the knowledge units and their relationships. The parsed content includes information such as document title, directory level, chapters, paragraphs, segments, tables, version number, effective date, and scope of application. Then, knowledge units such as entities, rules, conditions, indicators, actions, exceptions, and times are extracted to form the nodes of the knowledge graph. Relationships between these knowledge units, such as inclusion, referencing, aliasing, equivalence, update, validity period, applicability, exception, similarity, and evidence relationships, are extracted to form the edges of the knowledge graph.

[0144] Nodes in a knowledge graph are used to represent knowledge units within a knowledge base. Nodes can include, but are not limited to, the following types: document nodes (representing documents in the knowledge base); chapter nodes (representing chapters within a document); paragraph nodes (representing paragraphs within a document); chunk nodes (representing segments after a document has been divided); entity nodes (representing entities in the knowledge base, such as product names or department names); rule nodes (representing business rules or operational specifications); condition nodes (representing the preconditions or constraints for a rule); time version nodes (representing the effective date of a version); and table cell nodes (representing data units within a table).

[0145] Edges in a knowledge graph are used to represent the relationships between knowledge units. Edges can be of the following types, including but not limited to: containment edges, used to represent hierarchical relationships such as a document containing a chapter, a chapter containing a paragraph; reference edges, used to represent one knowledge unit referencing another knowledge unit; alias edges, used to represent two entities being aliases of each other; equivalent edges, used to represent two entities representing the same thing; update edges, used to represent a new version of a document updating an old version; validity period edges, used to represent a rule being valid for a specific time period; applicability edges, used to represent a rule applicable to a specific condition or entity; exception edges, used to represent a condition constituting an exception to a rule; similarity edges, used to represent two knowledge units with similar content but different applicable conditions or effects; and evidence edges, used to represent a knowledge unit as direct evidence for answering a certain type of question.

[0146] The knowledge graph G=(V,E) constructed in the above manner, where V represents the set of nodes and E represents the set of edges, fully represents the knowledge units in the knowledge base and the relationships between them.

[0147] S603. Based on the sampling constraints, obtain the subgraphs that satisfy the sampling constraints from the knowledge graph.

[0148] After obtaining the knowledge graph and determining the sampling constraints, a query is performed in the knowledge graph based on the sampling constraints to obtain the subgraphs that satisfy the sampling constraints.

[0149] Specifically, sampling constraints can be transformed into graph query conditions, including the types of nodes and edges to be retrieved, and the connections between nodes. Based on these query conditions, matching and retrieval are performed in the knowledge graph to obtain nodes and edges that meet the conditions. These nodes and edges constitute a subgraph. This subgraph contains knowledge units that precisely correspond to the sampling constraints and their associated relationships.

[0150] For example, for sampling constraints such as needing to obtain knowledge units of document type, needing to obtain knowledge units with update relationship and similarity relationship, needing to obtain the rule content of the current valid version as correct evidence, and needing to obtain the similar rule content of the obsolete version as interference evidence, the knowledge graph queries the nodes of document type, finds the correspondence between the old and new versions through update edges, finds rule nodes with similar content but different versions through similar edges, and obtains these nodes and edges to form a subgraph.

[0151] S604. Determine the subgraph as candidate information.

[0152] After obtaining a subgraph that satisfies the sampling constraints, this subgraph is identified as candidate information. This candidate information includes knowledge units obtained from the knowledge graph and their relationships, specifically comprising two parts: correct evidence and interfering evidence, which are used to generate test samples subsequently.

[0153] For example, the knowledge base stores multiple versions of car insurance policy documents, including version V2.0 (2023 version, abolished on December 31, 2024) and version V3.0 (2025 version, currently valid). This knowledge base is parsed to construct a knowledge graph: document nodes are created corresponding to V2.0 and V3.0 respectively; rule nodes are created corresponding to V2.0 rules and V3.0 rules respectively; time version nodes are created corresponding to the validity period of V2.0 (January 2023 to December 2024) and the validity period of V3.0 (from January 2025 onwards); update edges are created connecting V3.0 nodes and V2.0 nodes (indicating that V3.0 updates V2.0); similarity edges are created connecting V2.0 rule nodes and V3.0 rule nodes (indicating that two rules have similar but different content).

[0154] The candidate interaction logs show that when users asked questions using relative time expressions such as "this year," the system repeatedly recalled V2.0 documents and generated outdated answers. Based on these candidate interaction logs, the sampling constraints were determined as follows: document type nodes need to be obtained; nodes with update and similarity relationships need to be obtained; rule content from the currently valid version V3.0 needs to be obtained as correct evidence; and similar rule content from the obsolete version V2.0 needs to be obtained as interference evidence.

[0155] Based on the above sampling constraints, a query is performed in the knowledge graph. V3.0 and V2.0 document nodes are located in the knowledge graph. The update relationship between them is determined by updating edges, and the similarity relationship between two rule nodes is determined by similarity edges. The rule content corresponding to the V3.0 rule node is obtained as correct evidence, and the rule content corresponding to the V2.0 rule node is obtained as interference evidence. The nodes and edges obtained above constitute a subgraph containing the rules of the old and new versions and their relationships. This subgraph is identified as candidate information and used to subsequently generate test samples for the failure modes of relative time expression, multiple version documents, older versions being recalled more frequently, and answer expiration.

[0156] In this embodiment, within the scenario of a retrieval-enhanced generation system, a knowledge graph is constructed, and subgraphs satisfying the sampling constraints are extracted as candidate information. This allows the acquisition of candidate information to be based on the structured query of the knowledge graph, enabling accurate retrieval of correct and interfering evidence corresponding to the target path information from the knowledge base. The knowledge graph explicitly expresses semantic relationships such as updates, similarities, and references between knowledge units through nodes and edges, allowing the sampling constraints to perform accurate queries based on these semantic relationships. This avoids the semantic bias that may arise from keyword-based retrieval methods, providing accurate content material for the subsequent generation of targeted test samples.

[0157] Taking a search engine system as an example, when a user enters the query "how to maintain a lithium battery," the search engine retrieves relevant web pages from its webpage index and displays the search results. If the user repeatedly clicks on the search results, quickly returns, and changes their query to search again, this behavior can be considered feedback, indicating that the displayed search results did not meet the user's needs. The corresponding interaction record is then identified as a candidate interaction record. This candidate interaction record is analyzed to extract interaction path information that characterizes the search, such as "user uses fuzzy query terms, search engine retrieves low-authority web pages, high-quality web pages rank low, user does not click on any results and changes query terms." The frequency of each interaction path information in multiple candidate interaction records is counted, and interaction path information whose frequency meets the criteria is identified as the target path information, such as "user uses fuzzy query terms, low-authority web pages are retrieved, high-quality web pages rank low, user does not click." Based on this target path information, information matching the target path information is retrieved from the webpage index as candidate information, such as highly authoritative lithium battery maintenance guide web pages and low-authority personal experience sharing web pages, which serve as correct evidence and interfering evidence, respectively. Based on the above candidate information, a test sample is generated: the input information is "how to maintain a battery," the standard result information is "it is recommended to avoid overcharging and over-discharging, and to use the original charger," and the path identification information is "fuzzy query term, low authority recall, high-quality webpage ranking low, no clicks." When this test sample is used to test a search engine, if the search engine returns a low-authority personal experience sharing webpage, the path identification information can be used to quickly locate the unmet expected interaction pattern corresponding to the sample, without needing to re-analyze the reasons for the failure.

[0158] Taking the target system as an example of a search-enhanced RAG system, see [link to relevant documentation]. Figure 3 This illustrates a schematic diagram of the entire interactive process of a test sample generation method provided in an embodiment of this application. For example... Figure 3 As shown, the embodiments of this application involve the interaction between five participating entities: the user, the RAG system, the log collector, the processor, and the enhanced test sample.

[0159] During the operational phase, users interact with the RAG system through a question-and-answer process. Users input questions into the RAG system, which then performs retrieval, rearrangement, and generation processes before returning the answer to the user. Throughout this process, various modules of the RAG system automatically record logs. Users can also provide feedback on the system's returned answers, such as disliking or asking follow-up questions; this feedback is also recorded. A log collector gathers these scattered logs from the RAG system.

[0160] During the offline processing phase, the log collector delivers the collected logs to the processor. The processor first concatenates the scattered logs into complete interaction records, then filters out candidate interaction records belonging to the target interaction category (i.e., interaction records where the processing result does not meet expectations). Next, it extracts interaction path information from the candidate interaction records, determines the target path information based on the frequency of each interaction path information in the candidate interaction records, retrieves candidate information from the knowledge base based on the target path information, generates candidate test samples based on the candidate information and the target path information, verifies the candidate test samples, and uses the samples that pass the verification as test samples, ultimately forming an enhanced test sample set.

[0161] During the evaluation and optimization phase, enhanced test samples are used to evaluate the RAG system. The input information from the test samples is fed into the RAG system, which performs retrieval, rearrangement, and generation processes before returning an answer. The processor or evaluation module compares the returned answer with the standard results from the test samples and outputs an evaluation result. Based on the evaluation results, the RAG system is optimized in a targeted manner, and then re-evaluated. The optimized RAG system is then re-launched and continues logging. The processor analyzes the new logs, and if new high-risk paths are discovered, test samples for those paths are generated.

[0162] In another embodiment of this application, a readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the test sample generation method as described above.

[0163] In another embodiment of this application, an electronic device is also provided, see [link to relevant documentation]. Figure 4 The electronic device may include a processor 21 and a memory 22. The memory 22 stores executable instructions of the processor; the processor 21 is configured to execute the executable instructions to implement:

[0164] Acquire multiple interaction records of the target system, the interaction records including the recording information generated by the target system during the interaction process and the feedback information on the processing results;

[0165] Based on the feedback information in each interaction record, candidate interaction records belonging to the target interaction category are determined; wherein, the target interaction category is used to characterize that the processing result of the target system does not meet expectations;

[0166] Candidate information matching the candidate interaction record is determined from the information source corresponding to the target system;

[0167] Based on the candidate information, test samples are generated for testing the target system.

[0168] It should be noted that the specific implementation of the processor in this embodiment can be referred to the corresponding content above, and will not be described in detail here.

[0169] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0170] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0171] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0172] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating test samples, comprising: Acquire multiple interaction records of the target system, the interaction records including the recording information generated by the target system during the interaction process and the feedback information on the processing results; Based on the feedback information in each interaction record, candidate interaction records belonging to the target interaction category are determined; wherein, the target interaction category is used to characterize that the processing result of the target system does not meet expectations; Candidate information matching the candidate interaction record is determined from the information source corresponding to the target system; Based on the candidate information, test samples are generated for testing the target system.

2. The method according to claim 1, wherein acquiring multiple interaction records of the target system includes: Acquire the recorded information generated by the target system at multiple stages during the interaction process, as well as the feedback information on the processing results; wherein the stages are arranged in chronological order according to the interaction process of the target system, and include at least an input stage, a processing stage, and an output stage; The interaction record is obtained by associating the recorded information and feedback information belonging to the same interaction process.

3. The method according to claim 1, wherein determining the candidate information matching the candidate interaction record from the information source corresponding to the target system includes: Extract multiple interaction path information from the candidate interaction records; Count the frequency of occurrence of each interaction path information in each of the candidate interaction records; The frequency of occurrence represents the number of candidate interaction records that contain the interaction path information. The interaction path information whose frequency meets the frequency condition is determined as the target path information; Information that matches the target path information is obtained from the information source as candidate information.

4. The method according to claim 3, wherein determining the interaction path information whose occurrence frequency meets the frequency condition as the target path information includes: The interaction path information whose occurrence frequency is not less than the first threshold is determined as candidate path information; The candidate path information is determined as the target path information; or, The frequency of occurrence of the candidate path information in the first interaction record is obtained, where the first interaction record is the interaction record other than the candidate interaction record among the plurality of interaction records; The interaction path information whose frequency of occurrence in the first interaction record is not greater than the second threshold among the candidate path information is determined as the target path information.

5. The method according to claim 3, wherein the interaction path information is a combination of record information corresponding to multiple stages in the candidate interaction record, and the interaction path information includes at least one of the following types: It is composed of the recorded information corresponding to at least two consecutively arranged stages; It is composed of the recorded information corresponding to at least two non-contiguous stages; It consists of at least two stages of recorded information arranged in a specified chronological order.

6. The method according to claim 3, wherein determining candidate information matching the target path information from the information source corresponding to the target system includes: Based on the target path information, generate sampling constraints; Based on the sampling constraints, at least one information unit that satisfies the sampling constraints is obtained from the information source as the candidate information; The sampling constraints include at least one of the following: The type constraints of the information units involved in the target path information; The type of association relationship constraint between the information units; Extraction constraints for correct evidence used to generate correct samples. Extraction condition constraints for interference evidence used as interference items.

7. The method according to claim 1, wherein generating test samples for testing the target system based on the candidate information comprises: Based on the candidate information and target expression style information, input information and standard result information for candidate test samples are generated; Based on the target path information, path identification information of the candidate test sample is generated. The path identification information corresponds to the target path information and is used to characterize the unmet expected interaction mode corresponding to the candidate test sample. The candidate test samples are validated; The candidate test samples that pass the verification are used as the test samples.

8. The method according to claim 1, wherein, The target system is a retrieval enhancement generation system, and the interaction record is the log record information generated by the retrieval enhancement generation system at multiple stages during the processing of a question-and-answer interaction; the information source is a knowledge base.

9. The method according to claim 8, wherein determining candidate information matching the candidate interaction record from the information source corresponding to the target system includes: Based on the candidate interaction records, the sampling constraints are determined; Obtain the knowledge graph corresponding to the knowledge base. The knowledge graph includes nodes and edges. The nodes are used to represent knowledge units in the knowledge base, and the edges are used to represent the relationships between the knowledge units. Based on the sampling constraints, obtain a subgraph from the knowledge graph that satisfies the sampling constraints; The subgraph is identified as the candidate information.

10. An electronic device, comprising: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the executable instructions to achieve the following: Acquire multiple interaction records of the target system, the interaction records including the recording information generated by the target system during the interaction process and the feedback information on the processing results; Based on the feedback information in each interaction record, candidate interaction records belonging to the target interaction category are determined; wherein, the target interaction category is used to characterize that the processing result of the target system does not meet expectations; Candidate information matching the candidate interaction record is determined from the information source corresponding to the target system; Based on the candidate information, test samples are generated for testing the target system.

Images