Methods, apparatus, devices, storage media, and program products for generating dialogue test data.
By constructing a conversation graph and using a weighted random walk algorithm to generate dialogue test data, the problem of insufficient test coverage in existing technologies is solved, and efficient test data generation under automated and logical constraints is achieved.
Patent Information
- Application Number
- CN202611164928.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-03
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, the generation of test data for dialogue systems relies on human experience, resulting in insufficient test coverage and difficulty in covering all dialogue paths in complex business processes.
By constructing a conversation graph, a weighted random walk algorithm is used to generate multiple sequence paths. In-program and out-of-program nodes are used to represent core intentions and unexpected inputs to generate a dialogue test dataset.
It enables automated generation of test data, improving test coverage and generation efficiency, ensuring that the generated paths conform to business logic constraints, and excluding logically impossible paths.
Smart Images

Figure CN123114948A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium, and program product for generating dialogue test data. Background Technology
[0002] With the rapid development of large-scale language models and generative artificial intelligence technologies, the field of test data generation for dialogue systems has undergone a major transformation. Existing technical solutions are no longer limited to traditional manual coding methods, but increasingly rely on generative artificial intelligence models such as large-scale language models for automated test data generation.
[0003] In existing technologies, test scripts are usually based on fixed templates or manually written test scripts. Testers manually design dialogue processes and abnormal scenarios based on their business experience, write user inputs and corresponding expected system responses line by line, and form a dialogue test dataset.
[0004] However, this method has obvious drawbacks in practical applications. It mainly relies on the personal experience of testers. When the business process of the dialogue system is relatively complex, it is difficult for humans to cover all dialogue paths, resulting in a serious lack of test coverage. Summary of the Invention
[0005] Therefore, it is necessary to provide a method, apparatus, device, storage medium, and program product for generating dialogue test data to address the aforementioned technical problems.
[0006] Firstly, this application provides a method for generating dialogue test data, including:
[0007] Based on the business logic and dialogue flow of the dialogue system under test, a conversation graph is determined. The conversation graph includes in-program nodes and out-of-program nodes. In-program nodes are used to represent the core intent in the dialogue flow, while out-of-program nodes are used to represent unexpected inputs.
[0008] Based on the conversation graph, a weighted random walk algorithm is used to explore paths and generate multiple sequence paths; the sequence paths are used to represent the dialogue state transition sequence from the start node to the end node of the conversation graph.
[0009] Each sequence path is converted into a dialogue script to generate a dialogue test dataset.
[0010] In one embodiment, based on the session graph, a weighted random walk algorithm is used for path exploration to generate multiple sequence paths, including:
[0011] Select any node in the session graph as the starting node, and initialize the sequence path based on the starting node;
[0012] Starting from the starting node, calculate the transition probability based on the weights of each adjacent edge of the starting node, and determine the next node based on the transition probability and the weighted random walk algorithm. Record the next node in the sequence path, return to execute the step of determining the next node based on the transition probability and the weighted random walk algorithm, until the preset termination condition is met, stop jumping, and generate a sequence path.
[0013] Return to the step of selecting any node in the session graph as the starting node until the preset number of paths is reached, resulting in multiple sequence paths.
[0014] In one embodiment, determining the next node based on the transition probability and a weighted random walk algorithm includes:
[0015] Get all the neighboring nodes of the current node and the weights of the edges corresponding to each neighboring node, and randomly select a random number from the preset set of random numbers;
[0016] If the random number is less than the preset abnormal input probability and the current node is an in-program node, the abnormal injection mechanism is triggered to determine the next node from the in-program nodes of the current node.
[0017] If the random number is not less than the probability of abnormal input, calculate the transition probability of each node in the program based on the outgoing edge weight of each node in the program, and determine the next node from each node in the program based on the transition probability and the weighted random walk algorithm.
[0018] In one embodiment, identifying the next node from the out-of-program nodes of the current node includes:
[0019] Get the weights of the edges from the current node to each node outside the program;
[0020] The next node is determined from multiple out-of-program nodes based on their respective weights;
[0021] Among them, the higher the weight of an external node, the higher the probability of it being selected.
[0022] In one embodiment, the method for generating the dialogue test data further includes:
[0023] If the next node is an external node, a preset return strategy is used to determine the next node of the external node;
[0024] The return strategy involves setting the next node of the external node to the internal node before the jump.
[0025] In one embodiment, the method for generating the dialogue test data further includes:
[0026] If the next node is an external node, a preset transfer strategy is used to determine the next node of the external node;
[0027] The transition strategy involves determining the next jump node based on the weighted random walk algorithm, according to the outgoing edge weights of nodes outside the program.
[0028] Secondly, this application also provides an apparatus for generating dialogue test data, comprising:
[0029] The determination module is used to determine the conversation graph based on the business logic and conversation flow of the dialogue system under test. The conversation graph includes in-program nodes and out-of-program nodes. In-program nodes are used to represent the core intent in the conversation flow. Out-of-program nodes are used to represent unexpected input.
[0030] The exploration module is used to explore paths based on the conversation graph using a weighted random walk algorithm, generating multiple sequence paths; the sequence paths are used to represent the dialogue state transition sequence from the start node to the end node of the conversation graph;
[0031] The conversion module is used to convert each sequence path into a dialogue script and generate a dialogue test dataset.
[0032] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0033] Based on the business logic and dialogue flow of the dialogue system under test, a conversation graph is determined. The conversation graph includes in-program nodes and out-of-program nodes. In-program nodes are used to represent the core intent in the dialogue flow, while out-of-program nodes are used to represent unexpected inputs.
[0034] Based on the conversation graph, a weighted random walk algorithm is used to explore paths and generate multiple sequence paths; the sequence paths are used to represent the dialogue state transition sequence from the start node to the end node of the conversation graph.
[0035] Each sequence path is converted into a dialogue script to generate a dialogue test dataset.
[0036] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:
[0037] Based on the business logic and dialogue flow of the dialogue system under test, a conversation graph is determined. The conversation graph includes in-program nodes and out-of-program nodes. In-program nodes are used to represent the core intent in the dialogue flow, while out-of-program nodes are used to represent unexpected inputs.
[0038] Based on the conversation graph, a weighted random walk algorithm is used to explore paths and generate multiple sequence paths; the sequence paths are used to represent the dialogue state transition sequence from the start node to the end node of the conversation graph.
[0039] Each sequence path is converted into a dialogue script to generate a dialogue test dataset.
[0040] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:
[0041] Based on the business logic and dialogue flow of the dialogue system under test, a conversation graph is determined. The conversation graph includes in-program nodes and out-of-program nodes. In-program nodes are used to represent the core intent in the dialogue flow, while out-of-program nodes are used to represent unexpected inputs.
[0042] Based on the conversation graph, a weighted random walk algorithm is used to explore paths and generate multiple sequence paths; the sequence paths are used to represent the dialogue state transition sequence from the start node to the end node of the conversation graph.
[0043] Each sequence path is converted into a dialogue script to generate a dialogue test dataset.
[0044] The aforementioned method, apparatus, equipment, storage medium, and program product for generating dialogue test data determine a conversation graph based on the business logic and dialogue flow of the dialogue system under test. The conversation graph includes in-program nodes and out-of-program nodes. In-program nodes represent the core intent in the dialogue flow, while out-of-program nodes represent unexpected inputs. Based on the conversation graph, a weighted random walk algorithm is used for path exploration, generating multiple sequence paths. These sequence paths represent the dialogue state transition sequence from the start node to the end node of the conversation graph. Each sequence path is then converted into a dialogue script to generate a dialogue test dataset. This method solves the problem of insufficient test coverage in traditional methods. It also automatically excludes logically impossible paths, ensuring that the generated paths always conform to business logic constraints. Furthermore, it achieves end-to-end automated generation from path exploration to executable test cases, eliminating the need for manual test case writing and thus improving test coverage and generation efficiency. Attached Figure Description
[0045] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0046] Figure 1 This is an application environment diagram of a method for generating dialogue test data in one embodiment;
[0047] Figure 2 This is one of the flowcharts illustrating a method for generating dialogue test data in one embodiment;
[0048] Figure 3 This is a second flowchart illustrating a method for generating dialogue test data in one embodiment;
[0049] Figure 4 This is the third flowchart illustrating a method for generating dialogue test data in one embodiment;
[0050] Figure 5 This is a flowchart illustrating the method for generating dialogue test data in one embodiment (fourth example).
[0051] Figure 6 This is a structural block diagram of a device for generating dialogue test data in one embodiment. Detailed Implementation
[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0053] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.
[0054] With the rapid development of large-scale language models and generative artificial intelligence technologies, the field of test data generation for dialogue systems has undergone a major transformation. Existing technical solutions are no longer limited to traditional manual coding methods, but increasingly rely on generative artificial intelligence models such as large-scale language models for automated test data generation.
[0055] In existing technologies, test scripts are usually based on fixed templates or manually written test scripts. Testers manually design dialogue processes and abnormal scenarios based on their business experience, write user inputs and corresponding expected system responses line by line, and form a dialogue test dataset.
[0056] However, this method has obvious drawbacks in practical applications. It mainly relies on the personal experience of testers. When the business process of the dialogue system is relatively complex, it is difficult for humans to cover all dialogue paths, resulting in a serious lack of test coverage.
[0057] In view of the above-mentioned technical problems, this application provides a method for generating dialogue test data. The following embodiments will specifically illustrate the method for generating dialogue test data.
[0058] The method for generating dialogue test data provided in this application embodiment can be applied to, for example... Figure 1 The computer device shown includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores session graph data, sequence path data, and dialogue test datasets. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for generating dialogue test data.
[0059] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0060] In one exemplary embodiment, such as Figure 2 As shown, a method for generating dialogue test data is provided, which can be applied to... Figure 1 The following explanation uses computer equipment as an example, including:
[0061] S201. Determine the conversation graph based on the business logic and conversation flow of the dialogue system to be tested.
[0062] The conversation graph includes in-program nodes and out-of-program nodes; in-program nodes represent the core intent in the dialogue flow; out-of-program nodes represent unexpected inputs; this conversation graph is used to abstract all possible dialogue paths of the dialogue system under test into a directed weighted graph. .
[0063] In this embodiment, the computer device first acquires the business logic and dialogue flow of the dialogue system to be tested. The business logic includes at least the definition of the core business process, the temporal dependencies between each process node, and a set of business rules. The dialogue flow includes the interaction between the user and the dialogue system to be tested, as well as the user's intent, the dialogue system's response, and the legal sequence of transitions between the user's intent. Specifically, the computer device defines the core business process of the dialogue system to be tested as an in-program node. The temporal dependencies between process nodes are defined as directed edges between nodes within the program. ( For example, in an e-commerce intelligent customer service scenario, the core business process includes in-program nodes such as querying products, adding to cart, entering shipping address, generating order, payment confirmation, and order completion. The temporal dependencies are as follows: querying products can occur before or after adding to cart; entering shipping address must occur before generating order; payment confirmation must occur after generating order; and order completion must be the terminating node. If a transition violates this temporal dependency, such as jumping directly from querying products to payment confirmation, the directed edge will not exist in the session graph, or its weight will be set to zero, thereby achieving the construction of a constraint-aware session graph.
[0064] At the same time, computer devices define unexpected inputs or interruptions that users may make as out-of-program nodes. This includes casual conversation, asking about the weather, asking rhetorical questions, or expressing ambiguous intentions. Computer devices can determine the connection relationships between external and internal nodes based on the permissible scope of the business logic. For example, any internal node can be connected to chat or ask about the weather to simulate arbitrary interruptions by the user during task execution.
[0065] Furthermore, the computer equipment confirms each node based on business logic and dialogue flow. Directed edges between ) After that, weights can be assigned to each directed edge. The weight allocation can be done in any of the following ways: assigning weights based on the fundamental frequency of real user behavior to reflect the frequency of the most common behavioral paths in the real world; or assigning weights to each directed edge according to the test priorities pre-set by the test engineer. This is achieved by increasing the weights of paths leading to specific edge cases or critical failure nodes, ensuring the generation of sufficient targeted test cases. For example, the computer device can increase the weights of paths leading to complex authentication steps in a return process within the session graph, while setting the weights of edges that violate business logic to 0. Ultimately, the computer device generates a directed weighted graph G=(V, E, W) and uses it as the session graph, where V represents a node, E represents an edge, and W represents a weight.
[0066] S202, based on the conversation graph, uses a weighted random walk algorithm to explore paths and generate multiple sequence paths.
[0067] The sequence path is used to characterize the dialogue state transition sequence from the start node to the end node of the conversation graph.
[0068] In this embodiment, after constructing the conversation graph, the computer device randomly selects a starting node from the graph and selects the next node using a weighted random walk algorithm. Each selected node is recorded in the current path sequence until a termination condition is met, thus generating a complete sequence path. During node selection, the weights of directed edges control the tendency of the jump; edges with higher weights have a higher probability of being selected as the next node, thus achieving biased path exploration. Simultaneously, when the current node is an in-program node, the computer device triggers anomaly injection with a preset abnormal input probability, forcing a jump to an out-of-program node to simulate unexpected input or interruption behaviors that might occur in a real conversation. If the jump is to an out-of-program node, the computer device determines the subsequent path of that node according to a preset strategy. By repeatedly initiating the above path exploration process, the computer device generates multiple sequence paths covering different conversation state transition trajectories.
[0069] S203 converts each sequence path into a dialogue script, generating a dialogue test dataset.
[0070] In this embodiment of the application, the computer obtains each sequence path Then, the sequence path can be converted into a structured dialogue script that can be directly used for testing. Specifically, the computer device generates corresponding natural language text as user input based on the user intent and slot information represented by each node in each sequence path. For example, the computer device can call a large language model to assist in generating this natural language text to ensure the naturalness and diversity of the language. However, the underlying structure and logic of this natural language text are strictly controlled by the conversation graph and walk algorithm to ensure that it meets the semantic requirements of the user intent. Subsequently, the computer device generates dialogue data round by round according to the order of each node in the sequence path. Each round of dialogue data includes the user input, expected intent, expected system response, and dialogue state for that round. The computer device combines the dialogue data from each round in sequence to form a complete structured dialogue script. Finally, the computer device summarizes the structured dialogue scripts corresponding to each sequence path to generate a dialogue test dataset. This dialogue test dataset is output in standard JSON or YAML format and contains information such as dialogue rounds, user input, expected intent, expected system response, and dialogue state, which can be directly parsed and executed by automated testing frameworks.
[0071] The above method for generating dialogue test data involves determining a conversation graph based on the business logic and dialogue flow of the dialogue system under test. The conversation graph includes in-program nodes and out-of-program nodes. In-program nodes represent the core intents in the dialogue flow, while out-of-program nodes represent unexpected inputs. Based on the conversation graph, a weighted random walk algorithm is used for path exploration, generating multiple sequence paths. These sequence paths represent the dialogue state transition sequence from the start node to the end node of the conversation graph. Each sequence path is then converted into a dialogue script to generate the dialogue test dataset. This method solves the problem of insufficient test coverage in traditional methods. It also automatically excludes logically impossible paths, ensuring that the generated paths always conform to business logic constraints. Furthermore, it achieves end-to-end automated generation from path exploration to executable test cases, eliminating the need for manual test case writing and thus improving test coverage and generation efficiency.
[0072] In an exemplary embodiment, the phrase "based on the session graph, a weighted random walk algorithm is used to explore paths and generate multiple sequence paths" in S202 above, such as... Figure 3 As shown, it includes:
[0073] S301, Select any node in the session graph as the starting node, and initialize the sequence path based on the starting node.
[0074] In this embodiment, the computer device acquires a session graph and randomly selects a node from the node set V of the session graph as the starting node. The starting node can be either an in-program node or an out-of-program node. Subsequently, the computer device creates a sequence path S and uses this starting node as the first element of the path sequence, thus initializing the sequence path. Furthermore, the computer device will... (The sentence is incomplete and requires more context to translate accurately.) Set as the starting node ,Right now This is used as the starting point for subsequent iterative walks.
[0075] S302: Starting from the starting node, calculate the transition probability based on the weights of each adjacent edge of the starting node, determine the next node based on the transition probability and the weighted random walk algorithm, record the next node in the sequence path, return to execute the step of determining the next node based on the transition probability and the weighted random walk algorithm, until the preset termination condition is met, stop jumping, and generate a sequence path.
[0076] In this embodiment of the application, the computer device starts from the starting node. Starting from the initial node, the system begins path exploration. During path exploration, the computer first reads all neighboring nodes and their corresponding directed edge weights from the session graph. Then, it calculates the transition probability of each neighboring node based on the edge weights and selects the next node to jump to based on this probability using a weighted random walk algorithm. The selected next node is appended to the path sequence S. Further, the computer device updates this next node to the current node. The computer device repeats the above steps, navigating along the session graph. During each navigator, the computer device can also determine whether to trigger an abnormal navigator based on the current node type and preset anomaly injection conditions, in order to inject interference scenarios into the path sequence. When navigating to the preset termination node in the session graph... or the length of the path sequence S Reach the preset maximum number of dialogue rounds At this point, the computer equipment stops jumping, the current path exploration process ends, and a complete sequence path is finally generated.
[0077] S303, return to the step of selecting any node in the session graph as the starting node until the preset number of paths is reached, resulting in multiple sequence paths.
[0078] In this embodiment, the computer repeatedly executes the method described in S301, that is, each time it selects a node from the node set of the session graph as a starting node, and generates a complete sequence path based on the starting node, until the total number of generated sequence paths reaches the preset number of paths N, resulting in multiple sequence paths. During each path exploration process, the computer device starts from the starting node and explores the path based on a weighted walk algorithm. Each time a next node is determined, the computer device updates the current node to that next node, i.e., sets... And append the node to the path sequence S.
[0079] If the next node is determined at the current time If an abnormal injection occurs during the walk, the computer device uses a preset strategy to determine the next jump node for that external node. Specifically, the computer device can use either a return strategy or a transfer strategy to handle the external node. For the return strategy, the computer device sets the next node of the external node to the internal node before the jump, that is, returning to the business process node before the abnormal injection was triggered and continuing the original business process. This return strategy is used to simulate scenarios where a user resumes their original intention after a brief interruption, such as a user casually asking about the weather while shopping and then actively returning to the previous shopping process. This return strategy can be used to test whether the dialogue system can correctly maintain the dialogue context and resume the original business process after a brief interruption. For the transfer strategy, the computer device continues the walk based on the weights of the outgoing edges of the external node, using a weighted random walk algorithm. That is, starting from the current external node, the transfer probability is calculated according to the weights of its outgoing edges, and the next jump node is selected based on the transfer probability. This transfer strategy is used to simulate scenarios where the user's intention changes drastically, such as a user completely switching from shopping to casual conversation and no longer returning to the original business process. This transition strategy can be used to test the dialogue system's ability to handle a complete switch in user intent.
[0080] Computer equipment can preset which strategy to use, either a return strategy or a transfer strategy, based on the test requirements. Alternatively, it can dynamically select one strategy during the traversal based on the specific type of the current external node or the test objective.
[0081] The computer device repeatedly executes the above-described traversal and state update process until a preset termination condition is met, at which point it stops jumping and generates a complete sequence path. For example, the termination condition may include traversing to a preset termination node in the session graph. ; or the length of the path sequence S Reach the preset maximum number of dialogue rounds When any of the above conditions are met, the computer device can stop traversing the current path and output the current path sequence S. The computer device repeats the above path exploration process until the total number of generated sequence paths reaches the preset number of paths N, ultimately resulting in multiple sequence paths.
[0082] In an exemplary embodiment, the "determine the next node based on the transition probability and the weighted random walk algorithm" in S302 above, as shown in the example... Figure 4 As shown, it includes:
[0083] S401, obtain all neighboring nodes of the current node and the weights of the edges corresponding to each neighboring node, and randomly select a random number from the preset set of random numbers.
[0084] In this embodiment of the application, the computer device uses the current node As an index, query the set of all neighboring nodes of the current node from the session graph. And read the directed edge weights corresponding to each adjacent node. The adjacent nodes include nodes within the program and nodes outside the program. Then, the computer device randomly selects a random number from a preset set of random numbers. This is used to determine the trigger for subsequent anomaly injection mechanisms. The random number set is uniformly distributed within the interval [0,1), i.e. Each time a computer device performs a random walk, it independently samples a random number from this distribution.
[0085] S402, if the random number is less than the preset abnormal input probability and the current node is an in-program node, trigger the abnormal injection mechanism to determine the next node from the in-program nodes of the current node.
[0086] In this embodiment of the application, when the random number is less than the preset abnormal input probability and the current node For program nodes In the case that, and The computer device triggers an exception injection mechanism, which forces a jump from the current program node to an adjacent external program node. This anomaly injection mechanism is used to proactively introduce disturbances into core business processes to test the robustness and context preservation capabilities of the dialogue system under unexpected inputs. Compared to traditional methods where the occurrence of anomalies is random and uncontrollable, computer devices use preset probabilities of abnormal inputs. Precise control of the frequency of anomaly injection makes the coverage of anomaly scenarios systematic, quantifiable, and controllable.
[0087] Specifically, the computer device obtains the current node The weights of the edges to each external node are used to determine the next node from among the multiple external nodes. ( The greater the weight of an out-of-program node, the higher its probability of being selected. For example, if the current node is connected to both chat and weather inquiries out of the program, the computer device determines which out-of-program node to jump to based on the weight ratio of these two edges. In one implementation, the probability of abnormal input... It can be set to 5%. This jump process is not constrained by the regular transition probability and jumps directly to a node outside the program. It is used to simulate unexpected input or interruption behavior by the user during task execution.
[0088] S403: If the random number is not less than the probability of abnormal input, calculate the transition probability of each node in the program based on the outgoing edge weight of each node in the program, and determine the next node from each node in the program based on the transition probability and the weighted random walk algorithm.
[0089] In the embodiments of this application, in random numbers Not less than the probability of abnormal input In the case that, In other words, the computer device performs a regular walk. Specifically, the computer device obtains the outgoing edge weights from the current node to each node within the program, calculates the transition probability of each node based on these outgoing edge weights, and determines the next node from among the nodes within the program based on the transition probability. .
[0090] Specifically, for nodes Its distance to any adjacent node The transition probability can be determined by the edge weights. To make a decision, that is .in, It is a node The total strength, i.e., the strength of all slave nodes. The starting edge is the sum of its weights, and the sum of the transition probabilities of all in-process node neighbors is 1. The computer device randomly samples based on the calculated transition probability distribution to determine the next node.
[0091] Because the weights of all edges in the session graph that violate business logic are set to 0, the transition probability of adjacent nodes corresponding to edges with a weight of 0 is 0 when calculating transition probabilities, and they will not be selected during random sampling. Therefore, the computer device automatically eliminates all logically impossible paths during normal traversal, ensuring that the generated path sequence always conforms to the hard constraints of the business process. This logical constraint awareness mechanism allows the computer device to satisfy the temporal dependencies and rule constraints of the business logic without additional verification during path exploration.
[0092] In summary, based on all the above embodiments, a method for generating dialogue test data is also provided, such as... Figure 5 As shown, the method includes:
[0093] S501, Determine the conversation graph based on the business logic and conversation flow of the dialogue system to be tested;
[0094] S502, Select any node in the session graph as the starting node, and initialize the sequence path based on the starting node;
[0095] S503: Starting from the current node, obtain all adjacent nodes of the current node and the weights of the edges corresponding to each adjacent node, and randomly select a random number from the preset random number set; if the random number is less than the preset abnormal input probability and the current node is an in-program node, execute S504; if the random number is not less than the abnormal input probability, execute S505.
[0096] S504, if the random number is less than the preset abnormal input probability and the current node is an in-program node, trigger the abnormal injection mechanism to determine the next node from each out-of-program node of the current node.
[0097] S505: When the random number is not less than the probability of abnormal input, calculate the transition probability of each node in the program according to the weight of the outgoing edge of each node in the program, and determine the next node from each node in the program according to the transition probability and the weighted random walk algorithm.
[0098] S506, record the next node to the sequence path, and if the next node is an external node, use a return strategy or a transfer strategy to set the next node of the external node to the internal node before the jump;
[0099] S507, return to execute S503 until the preset termination condition is met, stop jumping, and generate a sequence path;
[0100] S508, return to execute S502, until the preset number of paths is reached, generating multiple sequence paths;
[0101] S509 converts each sequence path into a dialogue script, generating a dialogue test dataset.
[0102] The methods described in each of the above steps have been described in the foregoing embodiments. For details, please refer to the foregoing descriptions. They will not be repeated here.
[0103] The above embodiments are explained and illustrated by some examples below, which do not limit the technical solution.
[0104] In an exemplary application scenario, taking the e-commerce shopping process as an example, an e-commerce intelligent assistant is tested. Its core processes include: searching for products, adding to the shopping cart, checkout, and payment. The computer device constructs a session graph based on this business logic and dialogue flow, as shown in the graph parameter table. This session graph includes in-program nodes and out-of-program nodes to cover normal business processes and abnormal interruption scenarios. The parameter table for the session graph is shown in Table 1.
[0105] Table 1
[0106]
[0107] After constructing the session graph, the computer device assigns weights to each edge to control the bias of the traversal paths. In this example, the computer device sets a higher weight for the path N2 to N3 to N4 to N5 to N8 to simulate the high-frequency behavior path of real users; and sets a relatively high weight for the abnormal path N4 to N7 to focus on testing the robustness of the scenario where users ask for coupons at checkout. Simultaneously, the computer device sets the weight of edges that violate business logic to 0 to achieve logical constraint awareness, such as... This means you cannot skip the shopping cart and proceed directly to checkout; This indicates that items cannot be returned to the cart after payment. For example, the specific weight allocation is as follows: , , , as well as Abnormal redirect weight , Preset abnormal input probability .
[0108] Subsequently, the computer device explores the path starting from the initial node N1. Taking the current node N4 as an example, it randomly selects... The complete path generation process is shown in Table 2:
[0109] Table 2
[0110]
[0111] In round 4, the computer device is located at node N4 within the program and is randomly selected. ,because (0.10), triggering the exception injection mechanism, forcibly jumps from the current program node to its adjacent program node. The computer device obtains the weights of the edges from the current node N4 to each program node, where... , A secondary probability selection is performed based on the weights of each node, with N7 having a 70% probability of being selected and N6 having a 30% probability. Assuming N7 is selected in this random sampling, the node sequence is generated from N4 to N7, systematically creating a robust test scenario for querying coupons at checkout. In round 5, the computer device is located at node N7 outside the program. A return strategy is used to set the next node of this outside-program node as the in-program node N4 before the jump, simulating a scenario where the user briefly queries for coupons and then resumes their original intention to continue checking out.
[0112] Subsequently, the computer device continues to perform a regular walk, jumping from N4 to N5 and N8 in sequence according to the weight drive, completing the exploration of the entire path. The final generated complete path sequence is: [N1, N2, N3, N4, N7, N4, N5, N8].
[0113] The computer device converts the path sequence into a structured dialogue script, generating a partial test script, as shown in Table 3:
[0114] Table 3
[0115]
[0116] In this example, the computer device effectively ensures the legitimacy of core business processes by setting the weight of edges that violate business logic to 0, such as preventing users from skipping the shopping cart to checkout or returning items to the cart after payment. By pre-setting abnormal input probabilities, it systematically and controllably injects ambiguous intents or interfering information (AskCoupon) into the core processes, thereby effectively testing the dialogue model's ability to maintain and recover context. At the same time, through a weighted random walk algorithm, path exploration is biased towards high-frequency behavioral paths of real users and key business scenarios that testers focus on, achieving end-to-end automated generation from path exploration to executable test cases.
[0117] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.
[0118] Based on the same inventive concept, this application also provides a dialog test data generation apparatus for implementing the dialog test data generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more dialog test data generation apparatus embodiments provided below can be found in the limitations of the dialog test data generation method described above, and will not be repeated here.
[0119] In one exemplary embodiment, such as Figure 6 As shown, a device for generating dialogue test data is provided, comprising:
[0120] Module 11 is used to determine the conversation graph based on the business logic and conversation flow of the dialogue system under test. The conversation graph includes in-program nodes and out-of-program nodes. In-program nodes are used to represent the core intent in the conversation flow. Out-of-program nodes are used to represent unexpected input.
[0121] Exploration module 12 is used to explore paths based on the conversation graph using a weighted random walk algorithm, generating multiple sequence paths; the sequence paths are used to represent the dialogue state transition sequence from the start node to the end node of the conversation graph;
[0122] The conversion module 13 is used to convert each sequence path into a dialogue script and generate a dialogue test dataset.
[0123] In one embodiment, the exploration module 12 includes:
[0124] The selection unit is used to select any node in the session graph as the starting node and initialize the sequence path based on the starting node;
[0125] The calculation unit is used to start from the starting node, calculate the transition probability based on the weights of each adjacent edge of the starting node, determine the next node based on the transition probability and the weighted random walk algorithm, record the next node in the sequence path, return to execute the step of determining the next node based on the transition probability and the weighted random walk algorithm, until the preset termination condition is met, then stop jumping and generate a sequence path.
[0126] The return unit is used to return the step of selecting any node in the session graph as the starting node until the preset number of paths is reached, resulting in multiple sequence paths.
[0127] In one embodiment, the above-mentioned computing unit includes:
[0128] The random sub-unit is used to obtain all the neighboring nodes of the current node and the weights of the edges corresponding to each neighboring node, and to select a random number from a preset set of random numbers.
[0129] The exception subunit is used to trigger the exception injection mechanism when the random number is less than the preset exception input probability and the current node is an in-program node, and to identify the next node from each out-of-program node of the current node.
[0130] The probability subunit is used to calculate the transition probability of each node in the program based on the outgoing edge weight of each node in the program, provided that the random number is not less than the probability of abnormal input, and to determine the next node from each node in the program based on the transition probability and the weighted random walk algorithm.
[0131] In one embodiment, the above-mentioned abnormal subunit is specifically used for:
[0132] Get the weights of the edges from the current node to each node outside the program;
[0133] The next node is determined from multiple out-of-program nodes based on their respective weights;
[0134] Among them, the higher the weight of an external node, the higher the probability of it being selected.
[0135] In one embodiment, the above-mentioned computing unit further includes:
[0136] The first strategy subunit is used to determine the next node of the external node by adopting a preset return strategy when the next node is an external node; wherein, the return strategy is to set the next node of the external node as the internal node before the jump.
[0137] In one embodiment, the above-mentioned computing unit further includes:
[0138] The second strategy subunit is used to determine the next node of the external node by adopting a preset transfer strategy when the next node is an external node; wherein, the transfer strategy is to determine the next jump node based on the weighted random walk algorithm according to the outgoing edge weight of the external node.
[0139] Each module in the aforementioned dialog test data generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0140] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0141] Based on the business logic and dialogue flow of the dialogue system under test, a conversation graph is determined. The conversation graph includes in-program nodes and out-of-program nodes. In-program nodes are used to represent the core intent in the dialogue flow, while out-of-program nodes are used to represent unexpected inputs.
[0142] Based on the conversation graph, a weighted random walk algorithm is used to explore paths and generate multiple sequence paths; the sequence paths are used to represent the dialogue state transition sequence from the start node to the end node of the conversation graph.
[0143] Each sequence path is converted into a dialogue script to generate a dialogue test dataset.
[0144] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0145] Select any node in the session graph as the starting node, and initialize the sequence path based on the starting node;
[0146] Starting from the starting node, calculate the transition probability based on the weights of each adjacent edge of the starting node, and determine the next node based on the transition probability and the weighted random walk algorithm. Record the next node in the sequence path, return to execute the step of determining the next node based on the transition probability and the weighted random walk algorithm, until the preset termination condition is met, stop jumping, and generate a sequence path.
[0147] Return to the step of selecting any node in the session graph as the starting node until the preset number of paths is reached, resulting in multiple sequence paths.
[0148] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0149] Get all the neighboring nodes of the current node and the weights of the edges corresponding to each neighboring node, and randomly select a random number from the preset set of random numbers;
[0150] If the random number is less than the preset abnormal input probability and the current node is an in-program node, the abnormal injection mechanism is triggered to determine the next node from the in-program nodes of the current node.
[0151] If the random number is not less than the probability of abnormal input, calculate the transition probability of each node in the program based on the outgoing edge weight of each node in the program, and determine the next node from each node in the program based on the transition probability and the weighted random walk algorithm.
[0152] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0153] Get the weights of the edges from the current node to each node outside the program;
[0154] The next node is determined from multiple out-of-program nodes based on their respective weights;
[0155] Among them, the higher the weight of an external node, the higher the probability of it being selected.
[0156] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0157] If the next node is an external node, a preset return strategy is used to determine the next node of the external node;
[0158] The return strategy involves setting the next node of the external node to the internal node before the jump.
[0159] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0160] If the next node is an external node, a preset transfer strategy is used to determine the next node of the external node;
[0161] The transition strategy involves determining the next jump node based on the weighted random walk algorithm, according to the outgoing edge weights of nodes outside the program.
[0162] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0163] Based on the business logic and dialogue flow of the dialogue system under test, a conversation graph is determined. The conversation graph includes in-program nodes and out-of-program nodes. In-program nodes are used to represent the core intent in the dialogue flow, while out-of-program nodes are used to represent unexpected inputs.
[0164] Based on the conversation graph, a weighted random walk algorithm is used to explore paths and generate multiple sequence paths; the sequence paths are used to represent the dialogue state transition sequence from the start node to the end node of the conversation graph.
[0165] Each sequence path is converted into a dialogue script to generate a dialogue test dataset.
[0166] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0167] Select any node in the session graph as the starting node, and initialize the sequence path based on the starting node;
[0168] Starting from the starting node, calculate the transition probability based on the weights of each adjacent edge of the starting node, and determine the next node based on the transition probability and the weighted random walk algorithm. Record the next node in the sequence path, return to execute the step of determining the next node based on the transition probability and the weighted random walk algorithm, until the preset termination condition is met, stop jumping, and generate a sequence path.
[0169] Return to the step of selecting any node in the session graph as the starting node until the preset number of paths is reached, resulting in multiple sequence paths.
[0170] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0171] Get all the neighboring nodes of the current node and the weights of the edges corresponding to each neighboring node, and randomly select a random number from the preset set of random numbers;
[0172] If the random number is less than the preset abnormal input probability and the current node is an in-program node, the abnormal injection mechanism is triggered to determine the next node from the in-program nodes of the current node.
[0173] If the random number is not less than the probability of abnormal input, calculate the transition probability of each node in the program based on the outgoing edge weight of each node in the program, and determine the next node from each node in the program based on the transition probability and the weighted random walk algorithm.
[0174] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0175] Get the weights of the edges from the current node to each node outside the program;
[0176] The next node is determined from multiple out-of-program nodes based on their respective weights;
[0177] Among them, the higher the weight of an external node, the higher the probability of it being selected.
[0178] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0179] If the next node is an external node, a preset return strategy is used to determine the next node of the external node;
[0180] The return strategy involves setting the next node of the external node to the internal node before the jump.
[0181] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0182] If the next node is an external node, a preset transfer strategy is used to determine the next node of the external node;
[0183] The transition strategy involves determining the next jump node based on the weighted random walk algorithm, according to the outgoing edge weights of nodes outside the program.
[0184] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:
[0185] Based on the business logic and dialogue flow of the dialogue system under test, a conversation graph is determined. The conversation graph includes in-program nodes and out-of-program nodes. In-program nodes are used to represent the core intent in the dialogue flow, while out-of-program nodes are used to represent unexpected inputs.
[0186] Based on the conversation graph, a weighted random walk algorithm is used to explore paths and generate multiple sequence paths; the sequence paths are used to represent the dialogue state transition sequence from the start node to the end node of the conversation graph.
[0187] Each sequence path is converted into a dialogue script to generate a dialogue test dataset.
[0188] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0189] Select any node in the session graph as the starting node, and initialize the sequence path based on the starting node;
[0190] Starting from the starting node, calculate the transition probability based on the weights of each adjacent edge of the starting node, and determine the next node based on the transition probability and the weighted random walk algorithm. Record the next node in the sequence path, return to execute the step of determining the next node based on the transition probability and the weighted random walk algorithm, until the preset termination condition is met, stop jumping, and generate a sequence path.
[0191] Return to the step of selecting any node in the session graph as the starting node until the preset number of paths is reached, resulting in multiple sequence paths.
[0192] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0193] Get all the neighboring nodes of the current node and the weights of the edges corresponding to each neighboring node, and randomly select a random number from the preset set of random numbers;
[0194] If the random number is less than the preset abnormal input probability and the current node is an in-program node, the abnormal injection mechanism is triggered to determine the next node from the in-program nodes of the current node.
[0195] If the random number is not less than the probability of abnormal input, calculate the transition probability of each node in the program based on the outgoing edge weight of each node in the program, and determine the next node from each node in the program based on the transition probability and the weighted random walk algorithm.
[0196] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0197] Get the weights of the edges from the current node to each node outside the program;
[0198] The next node is determined from multiple out-of-program nodes based on their respective weights;
[0199] Among them, the higher the weight of an external node, the higher the probability of it being selected.
[0200] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0201] If the next node is an external node, a preset return strategy is used to determine the next node of the external node;
[0202] The return strategy involves setting the next node of the external node to the internal node before the jump.
[0203] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0204] If the next node is an external node, a preset transfer strategy is used to determine the next node of the external node;
[0205] The transition strategy involves determining the next jump node based on the weighted random walk algorithm, according to the outgoing edge weights of nodes outside the program.
[0206] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0207] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0208] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for generating dialogue test data, characterized in that, The method includes: Based on the business logic and dialogue flow of the dialogue system under test, a conversation graph is determined; the conversation graph includes in-program nodes and out-of-program nodes; the in-program nodes are used to represent the core intent in the dialogue flow; the out-of-program nodes are used to represent unexpected inputs; Based on the conversation graph, a weighted random walk algorithm is used to explore paths and generate multiple sequence paths; the sequence paths are used to represent the dialogue state transition sequence from the start node to the end node of the conversation graph. The sequence paths described are converted into dialogue scripts to generate a dialogue test dataset.
2. The method according to claim 1, characterized in that, Based on the session graph, a weighted random walk algorithm is used to explore paths and generate multiple sequence paths, including: Select any node in the session graph as the starting node, and initialize the sequence path based on the starting node; Starting from the starting node, the transition probability is calculated based on the weights of each adjacent edge of the starting node, and the next node is determined based on the transition probability and the weighted random walk algorithm. The next node is recorded in the sequence path, and the process of determining the next node based on the transition probability and the weighted random walk algorithm is repeated until a preset termination condition is met, at which point the jumping stops and a sequence path is generated. Return to the step of selecting any node in the session graph as the starting node, and continue until the preset number of paths is reached to obtain the multiple sequence paths.
3. The method according to claim 2, characterized in that, The step of determining the next node based on the transition probability and the weighted random walk algorithm includes: Obtain all neighboring nodes of the current node and the weights of the edges corresponding to each neighboring node, and randomly select a random number from a preset set of random numbers; If the random number is less than the preset abnormal input probability and the current node is an in-program node, an abnormal injection mechanism is triggered to identify the next node from each of the out-of-program nodes of the current node. If the random number is not less than the abnormal input probability, the transition probability of each node in the program is calculated based on the outgoing edge weight of each node in the program, and the next node is determined from each node in the program based on the transition probability and the weighted random walk algorithm.
4. The method according to claim 3, characterized in that, The step of confirming the next node from each of the out-of-program nodes of the current node includes: Obtain the weights of the edges from the current node to each of the external nodes; The next node is determined from the plurality of out-of-program nodes according to each of the aforementioned weights; Among them, the higher the weight of an external node, the higher the probability of it being selected.
5. The method according to claim 3, characterized in that, The method further includes: If the next node is an external node, a preset return strategy is used to determine the next node of the external node; The return strategy involves setting the next node of the external node as the internal node before the jump.
6. The method according to claim 3, characterized in that, The method further includes: If the next node is an external node, a preset transfer strategy is used to determine the next node of the external node. The transfer strategy involves determining the next jump node based on the weighted random walk algorithm, according to the outgoing edge weights of the nodes outside the program.
7. A device for generating dialogue test data, characterized in that, The device includes: The determination module is used to determine the conversation graph based on the business logic and conversation flow of the dialogue system under test; the conversation graph includes in-program nodes and out-of-program nodes; the in-program nodes are used to represent the core intent in the conversation flow; the out-of-program nodes are used to represent unexpected inputs; The exploration module is used to explore paths based on the conversation graph using a weighted random walk algorithm to generate multiple sequence paths; the sequence paths are used to represent the dialogue state transition sequence from the start node to the end node of the conversation graph. The conversion module is used to convert the sequence paths into dialogue scripts and generate a dialogue test dataset.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
