Test method, electronic device, and computer program product

Abstract

The present disclosure provides a test method, an electronic device and a computer program product, the method of the present disclosure first classifies the request content in the existing conversation record to obtain multiple request intents, then classifies the response content corresponding to the multiple request intents to obtain multiple response behaviors, a request intent and a response behavior corresponding to the request intent form a conversation mode, for different conversation modes, the test requirements applicable to different conversation modes are generated through the semantic features of the response content contained in the response behavior in the conversation mode, then the response service responds to the input of the conversation mode according to the changed prompt word to obtain the output response content, wherein the input of the conversation mode includes at least part of the request content corresponding to the conversation mode in the existing conversation record, finally, whether the changed prompt word passes the regression test is determined by determining whether the output response content meets the test requirements applicable to the corresponding conversation mode.

Description

Technical Field

[0001] This disclosure specifically relates to test methods, electronic devices, readable storage media, and computer program products. Background Technology

[0002] In modern business operations, intelligent interactive services can provide users with functions such as intelligent customer service, content generation, and data querying. Intelligent interactive services typically require prompts as the core of the business logic defining service behaviors (such as roles, rules, and output formats), and these prompts sometimes need to be modified to respond to market changes and feature upgrades.

[0003] However, changes to the prompts may lead to feature rollback; that is, the new rules expressed by the modified prompts may cause existing functions provided by the intelligent interaction service to malfunction. Therefore, it is necessary to test the modified prompts.

[0004] Currently, regression testing of prompts typically involves using a large amount of historical data to verify the modified prompts. This results in high testing costs and low testing efficiency. Summary of the Invention

[0005] This disclosure provides test methods, electronic devices, readable storage media, and computer program products.

[0006] The first aspect of this disclosure proposes a testing method, including: The request content in the existing session records is classified to obtain various request intents. The existing session records are generated by the response service responding according to the prompt words before the modification. The prompt words are configuration information used to define the behavior logic of the response service. The request content is the input content initiated by the user to the response service. The response content corresponding to the various request intentions is classified to obtain various response behaviors. The response content is the response result of the response service responding to the request content according to the original prompt words. The same request intention corresponds to one or more of the response behaviors. A request intention and a response behavior corresponding to the request intention form a session mode. For different session modes, the semantic features of the response content contained in the response behavior in the session mode are used to generate the test requirements applicable to each of the different session modes. The response service responds to the input of the session mode according to the modified prompt words, and the output response content is obtained. The input of the session mode includes at least part of the request content corresponding to the session mode in the existing session records. Whether the modified prompt word passes regression testing is determined by whether the response content of the output meets the test requirements applicable to the corresponding session mode.

[0007] According to some embodiments of this disclosure, the request content in existing session records is classified to obtain multiple request intentions, including: performing a first clustering on the request content in existing session records to obtain multiple request task categories; and performing a second clustering on the request content in each of the multiple request task categories to obtain multiple request intentions, wherein the request intentions corresponding to different request task categories are different.

[0008] According to some embodiments of this disclosure, before the response service responds to the input of the session mode according to the modified prompt words, the testing method further includes: The modified prompts are combined with the test requirements applicable to each of the different conversation modes, and the combination results are semantically understood to obtain the semantic understanding results. Based on the semantic understanding results, determine whether the instruction constraints contained in the modified prompt words semantically conflict with the rules contained in the test requirements; If so, the session pattern with the semantic conflict will be marked as a conflict path, a verification report will be generated, and the response service will no longer respond to the input of the session pattern according to the modified prompt words; The semantic understanding is performed using a language model with fewer parameters than the model used by the response service.

[0009] According to some embodiments of this disclosure, after obtaining multiple request intentions, the method further includes: adjusting the clustering structure for the request intentions whose clustering features meet the adjustment conditions.

[0010] According to some embodiments of this disclosure, adjusting the clustering structure of a request intent whose clustering features meet the adjustment conditions includes: splitting the request intent into multiple new request intents for request intents whose intra-class variance is greater than a variance threshold; and / or merging multiple request intents into a new request intent for request intents whose inter-class distance is less than a distance threshold.

[0011] According to some embodiments of this disclosure, generating test requirements applicable to different session modes by using the semantic features of the response content contained in the response behavior in the session mode includes: taking one or more of the semantic features that appear most frequently in the semantic features of the response content contained in the response behavior in the session mode as target semantic features, and generating test requirements applicable to different session modes based on the target semantic features, wherein the test requirements are used to indicate that the response content of the session mode needs to contain the content of the target semantic features.

[0012] According to some embodiments of this disclosure, determining whether the modified prompt word passes regression testing by determining whether the output response content meets the test requirements applicable to the corresponding session mode includes: determining the matching degree of the output response content to the test requirements applicable to the corresponding session mode; and determining whether the modified prompt word passes regression testing based on the matching degree and matching degree threshold of different session modes.

[0013] According to some embodiments of this disclosure, before the response service responds to the input according to the modified prompt word, the method further includes: determining the input corresponding to the session pattern by the semantic distance between the request content and the request intent contained in the existing session record corresponding to the request intent of the session pattern.

[0014] According to some embodiments of this disclosure, the input includes at least one of the request content that is semantically closest to or furthest from the cluster center of the request intent.

[0015] A second aspect of this disclosure provides an electronic device, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform the test method described in any of the above embodiments.

[0016] A third aspect of this disclosure provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the test method described in any of the above embodiments.

[0017] This disclosure provides a fourth aspect of a computer program product, the computer program product comprising a computer program, which, when executed by a processor, is used to implement the test method described in any of the above embodiments. Attached Figure Description

[0018] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0019] Figures 1 to 9 A flowchart illustrating a test method according to some embodiments of this disclosure is shown.

[0020] Figure 10 This is a schematic block diagram of the structure of a test apparatus according to one embodiment of the present disclosure.

[0021] Figure 11 This is a schematic block diagram of an electronic device 1000 according to one embodiment of the present disclosure. Detailed Implementation

[0022] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0023] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0025] The terminology used herein is for the purpose of describing particular embodiments and is not restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0026] Intelligent interactive services are used to interact with users, answering their questions, providing information they need, generating content they require, and offering various other functions. Intelligent interactive services can be implemented using language models, such as generative pre-trained transformers (GEMs). Language models use prompts to clarify their role in providing services, the rules they must follow, and other relevant information. As the interactive service is updated and iterated, the prompts will also be modified accordingly.

[0027] Due to the complexity of language models and the ambiguity of natural language, changes to prompts may have negative consequences. For example, a modified prompt might add rules related to information security, but this could lead to the incorrect rejection of legitimate core user requests, causing the language model to revert to its previous functionality. Therefore, modified prompts must be tested before being deployed online.

[0028] Currently, the safety of prompt word modifications is primarily ensured through full regression testing. Whenever a prompt word is modified, the entire historical data is used as input into the language model after the modification.

[0029] The above approach is costly because the business system has a large amount of historical data. Validating the entire dataset would incur significant token overhead, leading to high testing costs. Furthermore, iterative efficiency is low because validating large amounts of data takes a considerable amount of time, and developers cannot wait long for test results after each modification, resulting in slow technology iteration. Additionally, evaluating validation is difficult because testing with the entire dataset generates numerous new results, making it difficult to effectively assess their correctness using only manual or simple automated rules.

[0030] Therefore, this disclosure proposes a testing method.

[0031] Before developers begin testing the prompts, the user input (request content) in the historical conversation text is first semantically clustered to obtain N user intents. Then, the response content corresponding to the request content of each user intent is semantically clustered to obtain M response behaviors corresponding to the N user intents. A request intent and a corresponding response behavior constitute a conversation pattern (semantic path), resulting in P conversation patterns.

[0032] For each session mode, a sample request is selected from the request content belonging to the session mode as the test input, and the test requirements (behavioral specifications) are extracted from the response content belonging to the session mode. The test requirements represent the semantic content and statement structure of the expected output corresponding to the test input.

[0033] When testing the modified prompts, for each conversation mode, the sample request of the conversation mode is input into the language model A1 that responds using the modified prompts, and the response content corresponding to that conversation mode is output by model A1.

[0034] The response content and test requirements of the same conversational pattern are input into language model B1 for regression validation, resulting in P confidence scores for each conversational pattern output by model B1. The success of the regression test is determined based on these P confidence scores. Passing the test indicates that the modified prompts are usable. Failure to pass indicates that the modified prompts cause semantic content loss, semantic errors, or sentence structure discrepancies in the output of model A1. Using conversational pattern-based test inputs and requirements reduces both time and economic costs associated with testing.

[0035] Figure 1 A schematic diagram of the overall flow of the test method M100 according to some embodiments of this disclosure is shown. For example... Figure 2 The method shown includes steps S110, S120, S130, S140, and S150. This method can be executed by an electronic device such as a computer.

[0036] S110, classify the request content in the existing session records to obtain various request intents.

[0037] Among them, the existing session records are generated when the response service responds according to the original prompt words. The prompt words are configuration information used to define the behavior logic of the response service, and the request content is the input content initiated by the user to the response service.

[0038] The changes to the prompt words refer to modifications made by developers to the prompt words used to define the behavior logic of the response service, not modifications made by the user to the request content. This disclosure focuses on regression testing of the response service after the prompt words have been modified, to verify whether the modified prompt words cause a regression in the original functionality of the response service.

[0039] Existing session records can be obtained from historical session logs, or extracted from sample sessions input by developers and manually annotated corpus data.

[0040] The session log includes user-initiated requests. These requests can include user-entered text, which may represent questions, instructions, or statements. Different request contents can correspond to different request intentions.

[0041] Request intent is a category derived from the semantic abstraction and classification of request content. Different request intents correspond to different semantic directions, representing different user intentions. Request content within the same request intent exhibits a high degree of semantic similarity. For example, request intents can include question-and-answer intents, generation intents, and validation intents.

[0042] S120: The response content corresponding to various request intents is categorized to obtain various response behaviors. The response content is the result of the response service responding to the request content according to the original prompt. The same request intent corresponds to one or more response behaviors. A request intent and a corresponding response behavior form a session mode.

[0043] In addition to the request content, the session log also includes the response content of the interactive service in response to the request content. The response content is the reply generated by the interactive service based on the request content and the applied prompt words (the original prompt words), used to reply to the user. The request content and the corresponding response behavior can form a question-and-answer pair. Classified response content can all be correct response content; correspondingly, classified request content can all be request content with corresponding correct response content.

[0044] Each request intent contains at least one question-and-answer pair. For the response content within each question-and-answer pair of a request intent, classification can determine the response behavior type, resulting in one or more response behaviors. A response behavior represents a behavioral pattern, describing the response strategy adopted by the interactive service when faced with a specific request intent. For example, response behaviors may include success-generating, explanation-of-reason, and format-abnormal responses.

[0045] If the response content of a request intent is categorized and all of it falls into the same type of response behavior, then that request intent corresponds to only one type of response behavior. If the response content is categorized into different types of response behaviors, then that request intent corresponds to multiple types of response behaviors.

[0046] A conversational pattern represents an interactive scenario unit with an input-output relationship. It is a generalized representation of a class of similar interactions and is reusable.

[0047] In some implementations, after forming the session pattern, the method further includes: generating a summary for each session pattern and assigning a unique number. Specifically, the system automatically analyzes the keyword distribution, dependency structure, and topic characteristics of the clusters corresponding to the request content and / or response content in each session pattern, generates a general description as a summary of the session pattern, and assigns a unique number.

[0048] For example: Session Mode 01 (Path 01): Query order progress; Session Mode 02 (Path 02): Generates a Python script; Path 03 (Session Mode 03): User emotional soothing; Session Mode 04 (Path04): Table formatted output.

[0049] Through the above processing, each session pattern has a structured name and a traceable summary, so that the test requirements generated based on the session pattern, the semantic conflict judgment results in static verification, and the verification results of whether the regression test is passed after responding based on the modified prompt words can all correspond one-to-one with the session pattern, thereby realizing the structured tracking of the response behavior of the response service before and after the prompt words are modified.

[0050] For request intent q1, assuming the response content of intent q1 is classified into one response behavior x1, then q1 and x1 form a session pattern h1. For request intent q2, assuming the response content of intent q2 is classified into two response behaviors x2 and x3, then q2 and x2 form a session pattern h2, and q2 and x3 form a session pattern h3. Here, the request content in request intent q2 all represent the semantics of querying order progress, the response content in response behavior x2 all represent the semantics of needing more order information to query order progress, and the response content in response behavior x3 all represent the semantics of providing a structured format of order progress feedback.

[0051] S130, For different session modes, test requirements applicable to each session mode are generated by using the semantic features of the response content contained in the response behavior in the session mode.

[0052] The text of the response content has semantic features. For response content containing structured output, semantic features may include semantic fields, such as order ID and order status. For response content containing natural language output, semantic features may include semantic elements, such as semantic elements indicating that an order number is required, or semantic elements indicating apology and reassurance.

[0053] Test requirements are validation standards for a given session mode. They specify the semantic features or format constraints that the output of the interactive service must contain or meet under that mode. In other words, test requirements represent constraints on the response behavior of the interactive service. Test requirements can be expressed in natural language or structured formats. Test requirements allow us to test whether the difference between the output of the interactive service after a change in prompts and the output before the change is reasonable or acceptable.

[0054] For example, in the above session pattern h2, the intent q2 indicates querying order progress, and the intent q2 includes the request content j1: "I haven't received my order yet, can you help me check it?" For the response behavior x2 included in session pattern h2, the semantic features extracted from the response content of response behavior x2 include semantic elements y1 and y2, where y1 indicates a request for the user to provide the order number, and y2 indicates a commitment to query and provide feedback on the logistics location and estimated delivery time.

[0055] For the aforementioned session pattern h3, its intent q2 also includes the request content j2: "Please check the progress of the order with order number XXX." For the response behavior x3 included in session pattern h3, the semantic features extracted from the response content included in response behavior x3 include semantic fields such as order number field, logistics location field, and logistics location field.

[0056] Therefore, since the interaction scenarios of different session modes are different, the testing requirements are also different.

[0057] S140, the response service responds to the input of the session mode according to the modified prompt, and obtains the output response content. The input of the session mode includes the portion of the request content corresponding to the session mode from existing session records.

[0058] S150 determines whether the modified prompts pass regression testing by checking whether the output response content meets the test requirements applicable to the corresponding session mode.

[0059] The modified prompt words refer to the new prompt words used by the developers to optimize the interaction service (i.e., the response service). These new prompt words are the test objects in this implementation. A small number of request contents (requests previously initiated by the user) from existing session records are used as test requests. These test requests are input into the interaction service, and the output of the interaction service is obtained. The output of the interaction service is then evaluated according to the test requirements to determine whether the new prompt words disrupt existing functionality. The test requests can be randomly selected from the request content.

[0060] It should be noted that for the same user input, before the prompt word is changed, the existing session records already contain the output of the interaction service for that user input, referred to as the pre-change output. After the prompt word is changed, the output of the interaction service for that user input is obtained through step S140, i.e., the post-change output. The pre-change output and the post-change output are two different outputs for the same user input. Since the test requirements are extracted and formed from existing session records, that is, the standard session behavior is obtained by abstracting and organizing the session behavior before the prompt word is changed, evaluating the post-change output according to the test requirements can determine whether the post-change output meets the test requirements.

[0061] If the test requirements are met, it means that the modified output possesses the semantics and structure of standard conversational behavior. This indicates that the change in prompts has not weakened the interactive service's ability to respond to user input and can still respond and output as before the change. If the test requirements are not met, it means that the modified output lacks some or all of the semantics and structure of standard conversational behavior. This indicates that the change in prompts has weakened the interactive service's ability to respond to user input, disrupted the original functionality of the interactive service, and led to a functional regression.

[0062] The above testing process is performed for each session mode. The test requests used for each session mode are a small subset of the content requested in the existing session records for that session mode; for example, only one or two requests are selected from the corresponding request content for each session mode as test requests. Furthermore, in step S150, the output of the interactive service is evaluated using test requirements corresponding to the session mode.

[0063] The testing method proposed according to the embodiments of this disclosure does not directly perform full testing on massive historical data. Instead, it abstracts the dialogue history into session patterns containing requests and responses through semantic classification to represent a type of request and a type of response. It uses the response content of the response type in the session pattern to generate test requirements for each session pattern, thereby representing the elements required in the content of the interactive service response before the prompt word was changed. During testing, the same user input is used to obtain the output of the interactive service running with the modified prompt word, and the test requirements are used to evaluate the output, thereby determining whether the prompt word change is acceptable and whether it will cause the function to regress, thus realizing automatic consistency verification and behavior regression detection after the prompt word is modified.

[0064] Therefore, this implementation method only requires a small number of representative samples for testing, significantly reducing the number of test requests and token costs compared to full-scale testing. This makes continuous integration and agile iteration of prompt words possible in engineering, achieving high-coverage and low-cost verification after prompt word changes. Simultaneously, constructing session patterns through semantic classification can fully cover the semantic paths before and after prompt word changes. By establishing clear test requirements for each session pattern, the compliance of the interactive service's output can be determined at the semantic structure level. Compared to focusing only on the specific content of the output, this implementation method elevates regression testing from the phenomenological level to the logical level, and can predictably identify and avoid functional rollbacks caused by prompt word modifications, such as format corruption and intent deviation. This verifies whether prompt word changes disrupt the original behavior patterns, improving the stability and reliability of the interactive service.

[0065] Figure 2 This is a partial flowchart of one embodiment of the present disclosure.

[0066] refer to Figure 2 In some embodiments of this disclosure, before S140, when the response service responds to the input of the session mode according to the modified prompt, the test method M100 further includes: S202. Combine the modified prompts with the test requirements applicable to different conversation modes, and perform semantic understanding on the combination results to obtain the semantic understanding results. S204. Based on the semantic understanding results, determine whether the instruction constraints contained in the modified prompt words have a semantic conflict with the rules contained in the test requirements. S206. If so, mark the session mode with semantic conflict as a conflict path, generate a verification report, and no longer make the response service respond to the input of the session mode according to the modified prompt words.

[0067] Semantic understanding is performed using a language model with fewer parameters than the model used by the response service.

[0068] This implementation of static testing directly determines whether there is a semantic conflict between the content of the prompt and the content of the test requirements without invoking a response service. For example, the modified prompt and each test requirement can be combined to obtain the content to be verified. This content is then input into a language model, which performs semantic understanding and determines whether the instruction constraints in the prompt violate the rules in the test requirements. For instance, if the modified prompt contains the instruction constraint "do not output code," while a test requirement contains the semantic element "output should include a code block," a semantic conflict can be determined. If a semantic conflict is detected, it indicates that the modified prompt has a defect, and subsequent dynamic testing is unnecessary, thus avoiding the potential risk of feature rollback.

[0069] The semantic understanding described above can be performed using a lightweight language model with fewer parameters than the model used by the response service. Since static testing only requires semantic analysis and does not need to call the main model of the response service, a lightweight language model with fewer parameters can be used, further reducing testing costs and improving testing efficiency.

[0070] This implementation method employs a two-tiered mechanism of static testing and subsequent dynamic regression verification, enabling automatic consistency verification after the prompt words are modified.

[0071] Figure 3 A schematic flowchart of the test method M100 according to other embodiments of this disclosure is shown. (See also...) Figure 3In step S110, the method of classifying the request content in the existing session records to obtain multiple request intents can specifically include steps S111 and S112. Steps S111 and S112 use a two-layer clustering strategy to classify and obtain request intents.

[0072] S111, perform the first clustering on the request content in the existing session records to obtain multiple request task categories.

[0073] S112, perform a second clustering on the request content in various request task categories to obtain various request intents. Among them, the request intents corresponding to different request task categories are different.

[0074] The first clustering is a coarse-grained clustering method, used to divide the request content into various task categories according to semantic direction and density. For example, it can be divided into task categories such as question-and-answer (for question-and-answer), generation (for generation), and verification (for verification). The fundamental user intent corresponding to different task categories is different.

[0075] The second clustering is a finer-grained clustering based on the first clustering. It involves further clustering within each major category (semantic direction) obtained from the first clustering to obtain more specific task intents. The semantic consistency of the request content in the request intents obtained from the second clustering is higher than that of the request content in the task categories obtained from the first clustering. For example, the question-and-answer category can be divided into subcategories for order inquiries, price consultations, and address location, while the generation category can be divided into subcategories for code generation, copywriting generation, and summary generation. These subcategories constitute the request intents.

[0076] This implementation employs a two-layer clustering strategy. It effectively separates tasks with significant differences (such as question answering and generation) through coarse-grained clustering, while accurately identifying intent differences within a highly homogeneous semantic space through fine-grained clustering. This results in a request intent set with clearer semantics and more stable boundaries. Compared to single-level clustering, it balances the accuracy of macro-level task differentiation and micro-level intent recognition, improving the quality of conversation pattern construction and the accuracy of subsequent testing. Furthermore, this two-layer clustering strategy combines semantic embedding and output result clustering, constructing a conversation pattern (behavioral path) set that covers both input and output ends, thus providing a measurable and reusable test unit for subsequent verification.

[0077] Figure 4 A schematic flowchart of the test method M100 according to other embodiments of this disclosure is shown. (See also...) Figure 4After obtaining various request intents through step S110, step S119 is executed first to adaptively correct the clustering results of fine-grained clustering, and then step S120 is executed. In step S120, the response content is classified based on the request intents after adjusting the clustering structure.

[0078] S119, for the request intent where the clustering features meet the adjustment conditions, the clustering structure is adjusted.

[0079] Clustering features are quantitative metrics used to measure the internal tightness and external isolation of request intents as clusters. Adjustment conditions are preset threshold conditions used to determine whether clusters need to be split or merged. Cluster structure refers to the number of types of request intents and the request content they contain.

[0080] This implementation analyzes clustering features and dynamically adjusts request intents with excessively large intra-cluster differences (insufficient clustering) or excessively close cluster spacing (blurred boundaries), automatically optimizing the clustering structure to ensure that the semantics of each request intent are clear and the boundaries are stable, thus guaranteeing the effectiveness of the session mode.

[0081] Figure 5 A schematic flowchart of the test method M100 according to other embodiments of this disclosure is shown. (See also...) Figure 5 In step S119, the specific method for adjusting the clustering structure of the request intent for clustering features that meet the adjustment conditions may include one or more of steps S119a and S119b.

[0082] S119a, for request intents with intra-class variance greater than the variance threshold, split the request intent into multiple new request intents.

[0083] S119b: For multiple request intents with an inter-class distance less than a distance threshold, merge the multiple request intents into a new request intent.

[0084] Within-class variance is the sum of squared distances from the semantic vectors of all request content within a cluster to the centroid vector (semantic center) of that cluster. Semantic vectors are obtained by mapping natural language into fixed-dimensional vectors using semantic embedding algorithms, and the distance between semantic vectors represents the semantic similarity of natural language. Within-class variance is used to represent the dispersion of members within a cluster. The centroid vector can be the mean of the semantic vectors of all request content within the cluster. The distance between a semantic vector and the centroid vector can be Euclidean distance or cosine distance.

[0085] Inter-class distance is the distance between different clusters. It can be represented by the distance between the centroid vectors of the clusters or by the minimum distance between the clusters. Inter-class distance is used to represent the degree of separation between different clusters.

[0086] If the intra-class variance is greater than the variance threshold, it indicates that the intra-class variance is too large and the semantic consistency between the request content within the class is low. The cluster needs to be split into multiple sub-clusters, resulting in multiple sub-request intents, which are then added to the request intent set in place of the split request intents. If the inter-class distance is less than the distance threshold, it indicates that the inter-class distance is too small and the semantic consistency between the request content within multiple clusters is high. These clusters need to be merged into one cluster, which is then added to the request intent set in place of these clusters. If the intra-class variance of a cluster is not greater than the variance threshold, and the inter-class distances between this cluster and other clusters are all within the minimum inter-class distance threshold, then this cluster does not need to be split or merged.

[0087] This implementation splits a cluster with inconsistent internal semantics into multiple smaller clusters with more focused semantics, and merges multiple clusters with ambiguous semantic boundaries and similarity, thereby achieving fine-grained control over the clustering structure. This ensures that the set of request intents can cover multiple core scenarios while maintaining a high degree of consistency within each intent, thus enhancing the robustness and accuracy of the entire testing system.

[0088] In some implementations, the target content can be classified by clustering based on the textual features of the target content, using one or more of the request content and response content as the target content.

[0089] Text features can be semantic vectors. In steps S111 and S112, both the first clustering (coarse-grained clustering) and the second clustering (fine-grained clustering) can employ density-based clustering (such as the DBSCAN clustering algorithm), hierarchical clustering, or other clustering methods, thereby partitioning based on the density or distance of the semantic vector space. Regarding the ε representing the neighborhood radius and minPts representing the minimum number of neighbors applied when using the DBSCAN algorithm for clustering, the ε and minPts applied for coarse-grained clustering can both be higher than those applied for fine-grained clustering.

[0090] After obtaining the session patterns in step S120, the keywords, dependency structures, and topic features of the request intent in each session pattern can be determined through analysis, thereby generating a summary description as a summary of the session pattern and assigning a unique number to the session pattern. For example, a session pattern with the number 01 has the summary "query order progress".

[0091] Figure 6 A schematic flowchart of the test method M100 according to other embodiments of this disclosure is shown. (See also...) Figure 6In step S130, the method of generating test requirements applicable to different session modes by using the semantic features of the response content contained in the response behavior in the session mode can specifically include: taking one or more semantic features with the highest frequency of occurrence among the semantic features of the response content contained in the response behavior in the session mode as target semantic features, and generating test requirements applicable to different session modes based on the target semantic features. The test requirements indicate that the response content of the session mode must contain content with the target semantic features.

[0092] Each session pattern includes multiple question-answer pairs, which in turn include multiple response contents. These response contents belong to the same response behavior (i.e., the same session pattern), so they have many similar semantic features. For example, all response contents of a certain session pattern contain an order ID field, or all contain semantic tags indicating that an order number is required.

[0093] Target semantic features refer to one or more semantic features that appear most frequently in all response content of a conversational pattern. One way to extract target semantic features from response content is to input the response content into a pre-trained language model (such as the BERT model) to obtain the target semantic features.

[0094] Target semantic features can characterize the essential elements required for interactive services to respond in session mode. By using the most frequently occurring keywords or semantic fields as testing requirements, the response output of the interactive service under new prompts is considered to have retained key information only if it contains these keywords or semantic fields, thus ensuring functional consistency.

[0095] For example, in a certain session pattern, the request intent is "to check order progress," and the response behavior is information-seeking, i.e., asking the user for more information to complete the request intent. By extracting the semantic features of multiple response contents contained in this response behavior in this session pattern, the semantic element y1, which represents requesting the user to provide the order number, and the semantic element y2, which represents promising to query and provide feedback on the logistics location and estimated delivery time, are identified as having the highest frequency of occurrence. Each response content contains representations of y1 and y2. Therefore, the test requirement includes semantic elements y1 and y2.

[0096] It is understandable that semantic elements or keywords with a frequency higher than the frequency threshold can be used as target semantic features, or a specified number (e.g., 2) of the most frequently occurring semantic elements or keywords can be used as target semantic features.

[0097] This implementation method ensures consistency between the testing standard and historical high-quality responses by using the frequency of semantic features, and can identify the problem of missing key information caused by changes in prompt words.

[0098] In some implementations, the target semantic features include at least one of the target semantic fields and the target semantic tags.

[0099] Among them, target semantic fields are mainly applicable to structured or semi-structured response content, referring to specific data fields that must exist in the response. For example, "order_id" in a JSON response, and "order_id" in an XML response. <status>Tags, etc. For a session mode that requires generating a table, its target semantic features are mainly target semantic fields, such as the table must contain column names for product name, product price, and product quantity.

[0100] Target semantic tags are primarily applicable to unstructured natural language responses. They refer to specific semantic units or topics that must be included in the response. Examples include semantic elements or tags such as "order number required" or "suggestions for next steps." These tags can be extracted from the text using small models or rules. For a conversational pattern aimed at calming a customer, the target semantic features are mainly target semantic tags, such as the response needing to include semantic units like "expressing apology," "thanking for feedback," and "promising improvement."

[0101] This implementation method defines core semantic features through structured fields and unstructured tags, making the definition of test requirements more flexible and comprehensive. It can adapt to the diverse output formats generated in different business scenarios, improve the versatility of the test method, and achieve comprehensive coverage and accurate verification of different types of responses.

[0102] Figure 7 A schematic flowchart of the test method M100 according to other embodiments of this disclosure is shown. (See also...) Figure 7 In step S150, the modified prompt words are determined to pass the regression test by determining whether the output response content meets the test requirements applicable to the corresponding session mode. Specifically, this may include the following steps S151 and S152.

[0103] S151, Determine the degree to which the output response content matches the test requirements applicable to the corresponding session mode.

[0104] A session mode corresponds to a test requirement. A test requirement may contain one or more expected output elements. An expected output element corresponds to a semantic field, semantic tag, or keyword.

[0105] For expected output elements representing semantic fields, the matching score can be a binary result. For example, for each semantic field, the matching score is 1 if the response content contains the semantic field, and 0 if it does not. For expected output elements representing semantic tags or keywords, the matching score can be a similarity value ranging from [0,1]. For example, for each semantic tag in the test requirements, the matching score is calculated by evaluating the semantic similarity between the semantic tag and the response content.

[0106] Then, based on the matching degree of the expected output elements, the matching degree of the response content for the same test requirement is obtained, which is called the first matching degree, that is, the matching degree determined in step S151.

[0107] If all expected output elements in the same test requirement are represented by semantic fields, the first match score can be the product of the match scores of each expected output element. That is, if the response content does not contain the content of all expected output elements in the test requirement, the first match score of the response content for that test requirement is 0. The first match score is 1 only when the response content contains the content of all expected output elements in the test requirement. If there are expected output elements in the same test requirement represented by semantic tags or keywords, the first match score can be the average of the match scores of each expected output element.

[0108] Assuming there are P types of session modes, there are corresponding P types of test requirements. Each session mode has c inputs, so the response content output by the response service (interaction service) in step S140 has a total of P... c. P is obtained through step S151. The first match degree of each response content among c response contents is obtained as P. There are c first-degree matches, where each of the c responses is evaluated using the same set of test requirements.

[0109] Understandably, the first matching degree can be calculated by inputting the response content and corresponding test requirements into a finely tuned lightweight language model or a text embedding model connected to a classifier, and obtaining the first matching degree output by the model.

[0110] In addition to response content and corresponding test requirements, the data input to the aforementioned lightweight language model or text embedding model can also include triggering conditions. Triggering information provides context to the lightweight language model and text embedding model, thereby reducing the impact of noise on the model. Triggering information can be the request content or semantic features extracted from the request content (such as keywords or semantic tags). Triggering information helps the model understand the context, enabling it to more accurately calculate the first match.

[0111] S152, determine whether the modified prompt words pass the regression test based on the matching degree and matching degree threshold of different conversation modes.

[0112] After obtaining P After determining the first match score for c conversation patterns, the average of these first match scores can be used as the second match score, which is then compared to a preset match score threshold. If the second match score is greater than or equal to the match score threshold, the modified suggestion word is determined to have passed the regression test. If the second match score is less than the match score threshold, the modified suggestion word is determined to have failed the regression test.

[0113] Understandably, the pass rate of these first-match scores can also be used as the second-match score. The pass rate can be obtained by comparing each first-match score with a specified threshold. If a first-match score is greater than the specified threshold, the first-match score is considered to have passed; otherwise, it is considered to have failed. The quotient of the number of passed first-match scores to the total number of first-match scores is taken as the pass rate.

[0114] This implementation quantifies the degradation degree of interactive service responses by calculating the matching degree, making the regression test results more objective, detailed, and reliable, and effectively balancing the rigor of the test with the flexibility of the service output. Even if the response content of a certain input does not fully meet the test requirements, as long as the overall matching degree is higher than the matching degree threshold, the session mode can be considered stable. This makes the test results more flexible and practical, avoiding misjudgments caused by individual edge cases.

[0115] In some implementations, the test requirements may also include prohibited items. Prohibited items may include at least one of the following: unacceptable text content, unacceptable tone of voice.

[0116] Prohibited elements are used to specify elements that are forbidden from appearing in the response content. These can be the text content or the tone of the text. The text content in prohibited elements can include keywords or semantic tags such as "cannot answer" or "don't know" to prevent the interactive service from refusing to answer certain request intents. It can also include offensive language and other prohibited content. For some request intents, it is necessary to avoid responding with a specific tone. For example, for request content expressing anger, prohibited elements can include a tone that does not allow expressing anger to avoid giving users a bad experience.

[0117] Prohibited items can be generated by extracting semantic features from negative examples of the conversation pattern and generating them based on these semantic features. For example, historical conversation records may contain conversation records identified as negative examples. Conversation records that have been complained about or rated negatively by users can be identified as negative examples. Accordingly, the semantic features used in step S130 to generate test requirements can correspond to positive example conversation records. The method for extracting semantic features from negative examples can be the same as the method for extracting semantic features from positive examples.

[0118] P is determined through step S151. During the process of calculating c first-match scores, the first-match score can be calculated simultaneously based on the expected output elements (semantic fields, semantic tags, and keywords that need to appear) and prohibited items (semantic tags, keywords, sentence structure, and tone that are prohibited). If the response content contains any prohibited item, it means that the new prompt word reduces the quality of the interactive service output, so the first-match score can be determined to be zero, i.e., a complete mismatch.

[0119] For example, after calculating the product or mean of the matching scores of each expected output element, if it is determined that the response content contains any prohibited item, the first matching score is directly set to zero. Alternatively, if it is determined that the response content contains any prohibited item, the first matching score is directly set to zero before calculating the product or mean of the matching scores of each expected output element, and there is no need to calculate the product or mean of the matching scores of each expected output element.

[0120] Understandably, if a first match of zero is determined, the corresponding input request content, output response content, test requirements, prohibited items that cause the first match of zero, or expected output elements can be marked, and the marked content can be provided for error correction.

[0121] After obtaining the test results indicating whether the regression test passed or failed, a test report can be generated and provided based on the content of the aforementioned tags, the value of the first matching degree, and the value of the second matching degree. The report serves as a basis for revisions for developers, making the prompt word version change process observable, traceable, and verifiable.

[0122] Figure 8 A schematic flowchart of the test method M100 according to other embodiments of this disclosure is shown. (See also...) Figure 8 Before the response service responds to the input in the session mode according to the modified prompt (step S140), step S139 can be executed first. Step S139 corresponds to static test verification, while steps S140 and S150 belong to dynamic test verification.

[0123] S139, if the modified prompt contains instructions that are incompatible with the test requirements, a verification report is generated and the test is stopped. The verification report may include the part containing the incompatible prompt and the part containing the test requirements, and may also include an explanation of the reason for the incompatibility.

[0124] Static testing directly determines whether there is a semantic conflict between the content of the prompt and the content of the test requirements (such as expected output elements) without invoking interactive services. For example, the modified prompt and P test requirements can be combined to obtain P pieces of content to be verified. These P pieces of content are then input into a finely tuned lightweight language model or a text embedding model connected to a classifier. The model performs semantic understanding and determines whether the instruction constraints in the prompt violate the rules in the test requirements. For example, the modified prompt may contain the instruction constraint "do not output code," while a certain test requirement contains the semantic element "the output should contain a code block."

[0125] If the prompt word is semantically incompatible with the test requirements, a defect in the prompt word can be identified. Incompatible test requirements or corresponding session patterns can be marked, the reason for the incompatibility determined, and a verification report generated and provided to the user. Simultaneously, subsequent testing steps are stopped (static test verification failed), meaning steps S140 and S150 are not executed. If the prompt word is semantically compatible with the test requirements, the static test verification passes, and subsequent dynamic test verification can proceed.

[0126] This implementation method performs static testing before dynamic testing, and directly determines the overall test as failed if the static test fails, eliminating the need for subsequent dynamic testing and improving testing efficiency. Furthermore, static testing only performs semantic analysis and does not require service calls; the low token cost makes it suitable for quickly detecting potential risks during the prompt word development phase.

[0127] Figure 9 A schematic flowchart of the test method M100 according to other embodiments of this disclosure is shown. (See also...) Figure 9 Before the response service responds to the input according to the modified prompt (step S140), step S129 can be executed first. Step S129 is executed after step S120 and is used to clarify the specific request content to be input to the interaction service during dynamic testing.

[0128] S129, in the existing session records containing request content corresponding to the request intent of the session mode, determine the input corresponding to the session mode by the semantic distance between the request content and the request intent.

[0129] The test requests used in dynamic testing and verification only contain a small portion of the request content corresponding to the existing session patterns in the existing session records, rather than all request content. This eliminates the need for full regression testing, improving testing efficiency. Test requests can be determined by the vector distance (i.e., semantic distance) between the request content and the request intent in the semantic vector space. In other words, only request content that meets the semantic distance requirement is used as test requests. The smaller the distance from the centroid, the more typical the request; the larger the distance from the centroid, the more peripheral or ambiguous the request.

[0130] The semantic vector of a request intent can be the centroid vector of the request intent's cluster. The semantic distance is calculated as the distance between the semantic vector of each request content within the request intent and its centroid. For example, only one or more of the closest request content can be used as test input interaction services, and the modified prompt words can be determined to pass the test based on whether the output response content meets the corresponding test requirements.

[0131] This implementation discloses an input selection strategy for regression testing. Test requests are selected from the request content based on the vector distance between the request and the centroid vector. For example, requests close to the centroid can be selected as test requests that represent the intent of the request. This determines test requests with high coverage, good diversity, and small scale, thereby minimizing token consumption and test time while ensuring test effectiveness.

[0132] In some implementations, the input may include at least one of the request content that is semantically closest to or furthest from the cluster center of the request intent.

[0133] Among the multiple request contents contained in the request intent, the request content closest to the centroid vector of the request intent is the center sample, and the request content farthest from the centroid vector is the edge sample. A test request may include only one or more center samples, only one or more edge samples, or both. Both center samples and edge samples can be positive samples.

[0134] The test requests in this implementation can simultaneously cover both central and edge samples. The central samples are used to verify the functional correctness of the interactive service in standard scenarios, while the edge samples are used to verify the robustness and generalization ability of the interactive service in ambiguous or non-standard scenarios. This allows for a comprehensive evaluation of the impact of prompt word modifications, significantly reducing token consumption while ensuring coverage.

[0135] It is understandable that the selection of test requests can begin after obtaining the response behavior and session mode in step S120, selecting test requests from the corresponding request content for each session mode. Alternatively, it can begin after obtaining multiple request intents in step S110, determining the test requests for each request intent, for example, selecting a central sample and a marginal sample from the 200 request contents contained in intent q2. After obtaining the response behavior and session mode in step S120, the test requests can be updated based on the response behavior. For example, intent q2 corresponds to two response behaviors x2 and x3, where response behavior x2 corresponds to 60 of the 200 request contents, and response behavior x3 corresponds to 50 of the 140 request contents. Then, for the session mode formed by intent q2 and response behavior x2, the test requests are selected from the aforementioned 60 request contents. For the session mode formed by intent q2 and response behavior x3, the test requests are selected from the aforementioned 140 request contents.

[0136] The testing method disclosed in this embodiment can realize an automated closed loop from the modification of prompt words, static verification, dynamic testing to result backtracking, and achieve behavioral-level functional verification at a limited cost. Developers can get immediate feedback when submitting new versions, which significantly reduces the risk of function rollback and consistency issues caused by prompt word updates.

[0137] Based on any of the above embodiments, this disclosure also provides a testing apparatus. Figure 10 This is a schematic block diagram of the structure of a test apparatus according to one embodiment of this disclosure. Figure 10 As shown, the testing device includes: a request content classification module 110, a response content classification module 120, a test requirement generation module 130, a response module 140, and a test result determination module 150.

[0138] The request content classification module 110 is used to classify the request content in existing session records to obtain various request intents.

[0139] The response content classification module 120 is used to classify the response content corresponding to various request intents to obtain various response behaviors. The response content is the result of the response service responding to the request content according to the original prompt words. The same request intent corresponds to one or more response behaviors. A request intent and a corresponding response behavior form a session mode.

[0140] The test requirement generation module 130 is used to generate test requirements applicable to different session modes by using the semantic features of the response content contained in the response behavior in the session mode.

[0141] The response module 140 is used to enable the response service to respond to the input of the session mode according to the modified prompt words, and obtain the output response content. The input of the session mode includes the part of the request content corresponding to the session mode in the existing session records.

[0142] The test result determination module 150 is used to determine whether the modified prompt words pass the regression test by determining whether the output response content meets the test requirements applicable to the corresponding session mode.

[0143] The aforementioned testing device can be in the form of computer software, and each module of the testing device can be implemented through computer software modules. The specific implementation process of the functions and roles of each module in the aforementioned device is detailed in the corresponding steps of the above method, and will not be repeated here.

[0144] The subject executing the test method in the specific embodiments of this disclosure can be an electronic device such as a computer or server.

[0145] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the test method of any of the embodiments described above.

[0146] Figure 11 This is a schematic block diagram of an electronic device 1000 according to one embodiment of the present disclosure.

[0147] The hardware architecture of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0148] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0149] The processor 1200 can be a central processing unit (CPU). The processor 1200 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0150] The memory 1300 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions of a computer program in the embodiments of this disclosure. The processor 1200 implements the test method by running the non-transitory software programs, instructions, and modules stored in the memory 1300.

[0151] The memory 1300 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created by the processor 1200. Furthermore, the memory 1300 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1300 may optionally include memory remotely located relative to the processor 1200, and these remote memories may be connected to the processor 1200 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0152] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. "Readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of readable storage media include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0153] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the processes or functions of this disclosure are performed wholly or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, OAM, or other programmable device.

[0154] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0155] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0157] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0159] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0160] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0161] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.< / status>

Claims

1. A testing method, characterized in that, include: The request content in the existing session records is classified to obtain various request intents. The existing session records are generated by the response service responding according to the prompt words before the modification. The prompt words are configuration information used to define the behavior logic of the response service. The request content is the input content initiated by the user to the response service. The response content corresponding to the various request intentions is classified to obtain various response behaviors. The response content is the response result of the response service responding to the request content according to the original prompt words. The same request intention corresponds to one or more of the response behaviors. A request intention and a response behavior corresponding to the request intention form a session mode. For different session modes, the semantic features of the response content contained in the response behavior in the session mode are used to generate the test requirements applicable to each of the different session modes. The response service responds to the input of the session mode according to the modified prompt words, and the output response content is obtained. The input of the session mode includes at least part of the request content corresponding to the session mode in the existing session records. as well as Whether the modified prompt word passes regression testing is determined by whether the response content of the output meets the test requirements applicable to the corresponding session mode.

2. The test method according to claim 1, characterized in that, Before the response service responds to the input of the session mode according to the modified prompt, the testing method further includes: The modified prompts are combined with the test requirements applicable to each of the different conversation modes, and the combination results are semantically understood to obtain the semantic understanding results. Based on the semantic understanding results, determine whether the instruction constraints contained in the modified prompt words semantically conflict with the rules contained in the test requirements; If so, the session pattern with the semantic conflict will be marked as a conflict path, a verification report will be generated, and the response service will no longer respond to the input of the session pattern according to the modified prompt words; The semantic understanding is performed using a language model with fewer parameters than the model used by the response service.

3. The test method according to claim 1, characterized in that, The request content in existing session records is categorized to obtain various request intents, including: Perform a first clustering on the request content in the existing session records to obtain multiple request task categories; and The request content in the various request task categories is subjected to a second clustering to obtain various request intentions, and the request intentions corresponding to different request task categories are different.

4. The test method according to any one of claims 1 to 3, characterized in that, After obtaining multiple request intents, the method further includes: The request intent is to adjust the cluster structure for cluster features that meet the adjustment conditions; The request to adjust the clustering structure for clustering features that meet the adjustment conditions includes at least one of the following: For a request intent whose intra-class variance is greater than a variance threshold, the request intent is split into multiple new request intents; For multiple request intents where the inter-class distance is less than a distance threshold, the multiple request intents are merged into a new request intent.

5. The test method according to claim 1, characterized in that, The test requirements applicable to different session modes are generated by using the semantic features of the response content contained in the response behavior in the session mode, including: One or more of the semantic features that appear most frequently in the semantic features of the response content contained in the response behavior in the session mode are taken as target semantic features, and test requirements applicable to different session modes are generated based on the target semantic features. The test requirements are used to indicate that the response content of the session mode needs to contain the content of the target semantic features.

6. The test method according to claim 1 or 5, characterized in that, Whether the modified prompt words pass regression testing is determined by checking whether the response content of the output meets the test requirements applicable to the corresponding session mode, including: Determine the degree to which the output response content matches the test requirements applicable to the corresponding session mode; and Whether the modified prompt words pass the regression test is determined based on the matching degree and matching degree threshold of different conversation modes.

7. The test method according to claim 1, characterized in that, Before the response service responds to the input according to the modified prompt, the method further includes: In the existing session records containing request content corresponding to the request intent of the session pattern, the input corresponding to the session pattern is determined by the semantic distance between the request content and the request intent.

8. The test method according to claim 7, characterized in that, The input includes at least one of the request content that is semantically closest to and furthest from the cluster center of the request intent.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the test method according to any one of claims 1 to 8.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, is used to implement the test method according to any one of claims 1 to 8.

Images