Intelligent agent testing method and device, equipment, storage medium and program product

Abstract

The invention provides an intelligent agent testing method, and relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, the technical field of testing and the technical field of large models. According to the specific implementation scheme, input information and an interaction result for the input information are obtained, and a test data pair is obtained; combining the plurality of test data pairs with different intentions to obtain a test case; under the condition that it is determined that the test case meets the interaction logic standard, the test case is used for conducting multi-round interaction testing on the intelligent agent. The invention further provides an intelligent agent testing device, electronic equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the technical fields of human-computer interaction, deep learning, large models and testing, and specifically to a testing method, device, electronic device, storage medium and program product for an intelligent agent. Background Art

[0002] An agent is an agent that can perceive the environment and take actions to achieve specific goals. With the development of artificial intelligence technology, agents have been widely used, and are commonly seen in automation systems, robots, virtual assistants, etc. Before putting an agent into use, it needs to be tested to ensure that it can meet the use requirements when it is put into use. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device, storage medium and program product for testing an intelligent agent.

[0004] According to one aspect of the present disclosure, a method for testing an intelligent agent is provided, comprising: obtaining input information and interaction results for the input information to obtain test data pairs; combining a plurality of the test data pairs with different intentions to obtain test cases; and upon determining that the test cases meet the interaction logic standards, performing multiple rounds of interaction testing on the intelligent agent using the test cases.

[0005] According to another aspect of the present disclosure, a testing device for a pair of intelligent agents is provided, including: a data acquisition module, used to acquire input information and interaction results for the above input information to obtain a test data pair; a data combination module, used to combine a plurality of the above test data pairs with different intentions to obtain a test case; and an interaction test module, used to perform multiple rounds of interaction testing on the above intelligent agent using the above test case when it is determined that the above test case meets the interaction logic standard.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described above.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above method.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and the computer program implements the above method when executed by a processor.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 An exemplary system architecture that can be applied to the testing method and device for an intelligent agent according to an embodiment of the present disclosure is schematically shown;

[0012] Figure 2 A flowchart of a method for testing an intelligent agent according to an embodiment of the present disclosure is schematically shown;

[0013] Figure 3 The following schematically shows a flow chart of testing an intelligent agent according to an embodiment of the present disclosure;

[0014] Figure 4 A schematic diagram of generating a test data pair according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 5 Schematically shows a schematic diagram of rewriting input information according to an embodiment of the present disclosure;

[0016] Figure 6 Schematically shows a schematic diagram of generating test data pairs according to another embodiment of the present disclosure;

[0017] Figure 7 Schematically shows a schematic diagram of generating test interaction results according to an embodiment of the present disclosure;

[0018] Figure 8 A block diagram schematically shows a testing device for an intelligent agent according to an embodiment of the present disclosure; and

[0019] Fig. 9 A block diagram of an electronic device suitable for implementing a method for testing an intelligent agent according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] When using an intelligent agent, users usually express their usage requirements through natural language. Therefore, when testing the intelligent agent, testers are required to manually construct test cases in natural language based on their experience and use the test cases to test the intelligent agent.

[0022] However, the manual construction of test cases is inefficient, and due to the diversity and complexity of natural languages, manually constructed test cases are difficult to cover the usage scenarios of intelligent agents, which leads to low testing efficiency and difficulty in ensuring test quality.

[0023] Therefore, in order to improve the efficiency and accuracy of testing an intelligent agent, the embodiments of the present disclosure provide a testing method, device, electronic device and storage medium for an intelligent agent. The testing method for an intelligent agent includes: obtaining input information and interaction results for the input information to obtain a test data pair; combining multiple test data pairs with different intentions to obtain a test case; and when it is determined that the test case meets the interaction logic standard, using the test case to perform multiple rounds of interaction tests on the intelligent agent.

[0024] Figure 1 An exemplary system architecture to which the content processing method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0025] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the content processing method and apparatus can be applied may include a terminal device, but the terminal device may implement the content processing method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0026] like Figure 1As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0027] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0028] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0029] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0030] The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with blockchain.

[0031] It should be noted that the testing method for the intelligent agent provided in the embodiment of the present disclosure can generally be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, the testing device for the intelligent agent provided in the embodiment of the present disclosure can also be set in the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0032] Alternatively, the testing method for the intelligent agent provided in the embodiment of the present disclosure may also be generally performed by the server 105. Accordingly, the testing device for the intelligent agent provided in the embodiment of the present disclosure may generally be arranged in the server 105. The testing method for the intelligent agent provided in the embodiment of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the testing device for the intelligent agent provided in the embodiment of the present disclosure may also be arranged in a server or a server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0033] For example, a user can send a test instruction for the intelligent agent to the server 105 through the first terminal device 101, the second terminal device 102, and the third terminal device. The server 105 responds to the test instruction for the intelligent agent, obtains input information and interaction results for the input information, and obtains test data pairs; combines multiple test data pairs with different intentions to obtain test cases; and when it is determined that the test cases meet the interaction logic standards, uses the test cases to perform multiple rounds of interaction tests on the intelligent agent.

[0034] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0035] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0036] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0037] Figure 2 The flowchart of the method for testing an intelligent agent according to an embodiment of the present disclosure is schematically shown.

[0038] like Figure 2 As shown, the method includes operations S210 to S230.

[0039] In operation S210, input information and an interaction result for the input information are acquired to obtain a test data pair.

[0040] The input information may include information representing the interaction requirement, such as information (query) input by the user. The format of the input information is not limited, and may include, for example, natural language text, pictures, videos, or audio.

[0041] The interaction result for the input information may include an interaction reply obtained in response to the input information.

[0042] In some embodiments, a preset agent may be set to obtain an interaction result. The preset agent should be another agent other than the tested agent, specifically, another agent whose functions have been built or put into use. Input information may be input to the preset agent, and the preset agent generates an output result for the input information. For example, the preset agent may obtain an answer based on its own function as an interaction result, or the preset agent may call an external tool and obtain an interaction result using the external tool.

[0043] In other embodiments, probe technology can be used to attach a traffic recording agent to the human-computer interaction process. The traffic recording agent can use bytecode enhancement technology to insert codes for obtaining input information and interaction results into the human-computer interaction system to facilitate obtaining input information and interaction results in the human-computer interaction process.

[0044] The test data pair may include input information and an interaction result for the input information, or may include input information with the same intent as the input information and an interaction result for the input information. Specifically, the test data pair may be in a lightweight data exchange format (JavaScript Object Notation, JSON).

[0045] Since different users have different usage requirements and language expression habits, the input information obtained based on user input is diverse and can cover more comprehensive scenarios. In addition, by directly obtaining the input information and the interaction results for the input information, test data pairs are obtained, without the need for testers to manually construct test data for the agent, thus improving test efficiency.

[0046] In operation S220, a plurality of test data pairs with different intentions are combined to obtain a test case.

[0047] Intent can be used to represent the use requirements of the agent. Since users usually express their use requirements to the agent by inputting information, the intent of the input information can be identified based on the input information of the test data pair to determine the intent of the test data pair.

[0048] In some embodiments, the input information may be identified by matching. Specifically, keywords in the input information may be extracted, the keywords may be matched with the intents in the preset intent library, and the intent with the highest matching degree with the keywords may be determined as the intent of the input information.

[0049] In other embodiments, a classification model may be used to identify the intent of the input information. Specifically, the input information may be input into the classification model, and the classification model classifies the intent according to the features of the input information to determine the intent of the input information.

[0050] A test case may include multiple test data pairs with different intentions. When combining multiple test data pairs, the timing of the multiple test data pairs may be determined according to the respective intentions of the multiple test data pairs, and then the multiple test data pairs may be combined according to the timing.

[0051] Before combining multiple test data pairs, the test data pairs may be screened to select test data pairs with higher quality as the test data pairs to be combined.

[0052] Since the agent usually depends on the context of interaction with the user when giving the interaction result, it is difficult to test the context learning ability of the agent when constructing the test case only by a single test data pair. The embodiment of the present disclosure can make the test case test the context learning ability of the agent by combining multiple test data pairs with different intentions.

[0053] In addition, different intentions can be understood as different tasks performed or functions used for input information. For example, the intention of test data pair A is to order takeout, and the intention of test data pair B is to open the music player to play the song of singer AA. By combining multiple test data pairs with different intentions, different types of tasks or functions can be combined, making the entire test case diverse and complex, and covering multiple usage scenarios of the agent at the same time.

[0054] In operation S230, when it is determined that the test case meets the interaction logic standard, the test case is used to perform multiple rounds of interaction tests on the agent.

[0055] The interactive logic standard can be used to evaluate the interactive logic of test cases. When testing an agent with a test case that does not meet the interactive logic standard, it is impossible to truly simulate the user's real interactive scenario, and therefore it is impossible to accurately test the agent's real capabilities, resulting in inaccurate test evaluation results. Therefore, before using a test case to test an agent, it is possible to determine whether the test case meets the interactive logic standard to ensure the quality of the test case.

[0056] When determining whether a test case meets the interaction logic standard, it can be determined whether multiple test data pairs in the test case meet the interaction logic standard, or it can also be determined whether the input information and the interaction result for the input information in the test data pair meet the interaction logic standard.

[0057] In some embodiments, test cases that do not meet the interaction logic standard may be discarded to prevent them from affecting the accuracy of the test.

[0058] In other embodiments, test cases that do not meet the interaction logic standard may also be adjusted to meet the interaction logic standard.

[0059] The test of performing multiple rounds of interactions on an intelligent agent using a test case may include: using each test data pair to interact with the intelligent agent in turn according to the time sequence of multiple test data pairs. Specifically, input information in the first test data pair is input to the intelligent agent, a first round of interaction is performed, and the test interaction result of the intelligent agent in the first round of interaction is obtained, and then input information in the second test data pair is input to the intelligent agent, a second round of interaction is performed, and the test interaction result of the intelligent agent in the second round of interaction is obtained, ..., until input information in the last test data pair is input to the intelligent agent, a last round of interaction is performed, and the test interaction result of the intelligent agent in the last round of interaction is obtained, and multiple rounds of interaction are completed to obtain the test interaction results of the intelligent agent in multiple rounds of interaction.

[0060] In some embodiments, the test interaction results generated by the agent can be evaluated using the interaction results for the input information in the test data pair in each round of interaction to obtain the sub-test evaluation results of the current round. Then, based on the respective sub-test evaluation results of each round, the test evaluation results of the agent under the test case are obtained. Specifically, based on the respective intentions of each test data pair in the test case, weights can be set for multiple test data pairs respectively, and based on the weight of each test data pair and the sub-test evaluation results, the test evaluation results of the agent under the test case are obtained.

[0061] In other embodiments, the interaction results of multiple test data pairs for input information can be spliced ​​together to obtain expected results, and the test interaction results of the agent in multiple rounds of interaction can be spliced ​​together to obtain target test interaction results, and based on the similarity between the expected results and the target test interaction results, the test evaluation results of the agent under the test case can be determined.

[0062] In the disclosed embodiment, since the test data pairs are obtained from the real interaction records of the preset intelligent agent, the test data pairs can not only characterize the real usage requirements of the intelligent agent, but also have richness and complexity, and also increase the efficiency of constructing the test data. Moreover, by combining multiple test data pairs with different intentions to obtain test cases, not only the obtained test cases can test the contextual learning ability of the intelligent agent, but also the complexity of the test cases is increased, and more usage scenarios can be covered. In addition, by selecting test cases that meet the interactive logic standards, the intelligent agent is tested to further improve the availability of the test cases. Thus, by using test cases with richness, comprehensiveness, complexity and high availability to perform multiple rounds of interactive testing on the intelligent agent, the testing efficiency and quality of the intelligent agent are improved.

[0063] Reference below Figure 3~Figure 6 , combined with specific embodiments Figure 2 The method shown is further explained.

[0064] Figure 3 The figure schematically shows a flow chart of agent testing according to an embodiment of the present disclosure.

[0065] like Figure 3 As shown, in the scenario of testing the intelligent agent in embodiment 300, input information and interaction results for the input information can be obtained, such as input information 301_1 and interaction result 302_1, input information 301_2 and interaction result 302_2, ..., input information 301_P and interaction result 302_P.

[0066] Test data pairs can be obtained based on input information and interaction results. For example, based on input information 301_1 and interaction result 302_1, test data pair 303_1 can be obtained; based on input information 301_2 and interaction result 302_2, test data pair 303_2 can be obtained; and based on input information 301_P and interaction result 302_P, test data pair 303_P can be obtained.

[0067] A plurality of test data pairs with different intentions may be combined to obtain a test case 304, and the test case 304 may be used to test the agent M1.

[0068] According to the embodiments of the present disclosure, Figure 2 The operation S220 shown, obtaining input information and interaction results for the input information to obtain a test data pair, may include: extracting target input information from multiple intention sets with different intentions, and obtaining a test data pair based on the target input information and the interaction results.

[0069] The intent set may include input information or rewritten input information. The rewritten input information is obtained by rewriting the input information using a large model. Multiple input information in the intent set represent the same intent, that is, the input information and the rewritten input information may have the same intent. The target input information may be the input information in the intent set or the rewritten input information in the intent set.

[0070] The input information can be rewritten using a large model to obtain multiple rewritten input information while ensuring that the intent remains unchanged, and an intent set can be obtained based on the input information and the rewritten input information. Specifically, the words in each input information can be rewritten with synonyms, or the word order of the input information can be adjusted to obtain the rewritten input information.

[0071] When the input information is rewritten using the large model, a prompt word for rewriting can be input into the large model. Specifically, the prompt word for rewriting can include the intention of the input information and the rewriting requirement.

[0072] For example, the prompt words input to the large model can be "You are an intelligent agent rewriting expert. Please rewrite the input information and get two rewritten input information. The requirements are as follows: 1. The intention of the rewritten input information must remain unchanged; 2. The length of the rewritten input information must not exceed 20 Chinese characters; 3. The rewritten input information does not contain itself; 4. The rewritten input information is output in JSON format."

[0073] The input information may be "change the ad bid to 20 yuan". After rewriting the input information, the large model may obtain the rewritten input information such as "set the bid to 20 yuan", "please set my ad bid to 20 yuan", and output it in JSON format as {"change the ad bid to 20 yuan": ["set the bid to 20 yuan", "please set my ad bid to 20 yuan"]}. The final intent set includes "change the ad bid to 20 yuan", "set the bid to 20 yuan", and "please set my ad bid to 20 yuan".

[0074] In an embodiment of the present disclosure, a test data pair may include target input information and an interaction result. Since the target input information is extracted from an intent set, and the intent of the target input information is the same as the intent of the input information, the interaction result for the input information may also be the interaction result for the target input information, and a test data pair may be obtained based on the target input information and the interaction result for the input information.

[0075] The embodiments of the present disclosure expand test data by rewriting input information, thereby improving the diversity and richness of test cases.

[0076] Figure 4 The diagram schematically shows a schematic diagram of generating test data pairs according to an embodiment of the present disclosure.

[0077] like Figure 4 As shown, in embodiment 400, input information 401 may be input into a large model M2, and the input information may be rewritten using the large model M2 to obtain rewritten input information 402_2, and based on the input information 401 and the rewritten input information 402_1, an intention set 402 may be obtained.

[0078] Target input information 403 is obtained from the intention set 402 , and based on the target input information 403 and the interaction result 404 , a test data pair 405 is obtained.

[0079] According to an embodiment of the present disclosure, rewriting the input information may include: extracting key fields from the input information, and semantically rewriting the input information using a large model while retaining the key fields to obtain rewritten input information.

[0080] The key fields may be fields in the input information that are related to the intent. Since the large model may not be able to accurately identify the intent represented by the input information when rewriting the input information using the large model, the key fields in the input information may be determined before the input information is input into the large model for rewriting, and the large model may be prompted to retain the key fields for rewriting so that the intent of the rewritten input information remains unchanged.

[0081] For example, in a scenario of modifying an advertisement bid, the key field may be the advertisement price, that is, when the input information is "modify the advertisement bid to 20 yuan", the key field extracted from the input information may be "20 yuan".

[0082] When retaining key fields and semantically rewriting input information using a large model, a prompt for retaining key fields can be added to the prompt words used for rewriting. For example, in the scenario of modifying an advertisement bid, when the input information is "modify the advertisement bid to 20 yuan", the prompt for retaining key fields can be: retain "20 yuan" in the input information and semantically rewrite the input information.

[0083] Since the key fields are retained in the rewritten input information, and the rewritten input information is obtained by semantically rewriting the input information, the rewritten input information has the same intention as the input information.

[0084] The embodiments of the present disclosure semantically rewrite the input information by retaining key fields, thereby ensuring that the intention of the rewritten input information remains unchanged while ensuring the retention of key field information, thereby ensuring the correlation between the rewritten input information and the input information, as well as the correlation between the rewritten input information and the interaction results, thereby improving test accuracy.

[0085] Figure 5 A schematic diagram of rewriting input information according to an embodiment of the present disclosure is schematically shown.

[0086] like Figure 5 As shown, in embodiment 500, key fields 503 can be extracted from input information 501. When input information 501 is input to large model M2 for rewriting, large model M2 can be prompted to retain key fields 502 in input information 501, and multiple rewritten input information 503 are obtained.

[0087] According to an embodiment of the present disclosure, after performing operation S220 as shown in the figure, Figure 2 Before the operation S230 shown, the testing method for the intelligent agent may further include: performing logic compliance identification on the test case to obtain a compliance identification result. When the compliance identification result indicates that the test case is compliant, determining that the test case meets the interaction logic standard.

[0088] Logical compliance identification can be used to identify whether a test case meets the interactive logic standard. Since a test case is obtained by combining multiple target input information and the interactive results of the input information, the obtained test case may not meet the interactive logic standard. In order to improve the quality of the test case, the test case can be logically identified before the test case is used to test the agent, so as to ensure that the test case that meets the interactive logic standard is used for testing, thereby ensuring the test quality.

[0089] Test case compliance can be characterized by the fact that each test data pair within the test case is logically consistent, and the execution order of multiple test data pairs is logically consistent with each other.

[0090] When identifying the logical compliance of a test case, you can first determine whether the logic of the test data pair is compliant, and then determine whether the logic between multiple test data pairs is compliant. Specifically, if each test data pair is logically compliant and multiple test data pairs are logically compliant, the test case is determined to be compliant. If there are illogical test data pairs, or if multiple test data pairs are illogical, the test case is determined to be non-compliant.

[0091] The embodiments of the present disclosure can ensure the logic of test cases by performing compliance identification on test cases, thereby improving the accuracy of the test.

[0092] According to an embodiment of the present disclosure, logical compliance identification is performed on the test case to obtain a compliance identification result, including at least one of the following: compliance identification of the execution order of multiple intentions of the test case; rewriting compliance identification of the rewritten input information in the test case; and association compliance identification of the rewritten input information and the interaction result.

[0093] Since a test case is composed of test data pairs of multiple intents, and multiple test data pairs have an execution order, it is necessary to identify the compliance of the execution order of multiple intents to avoid illogicality between multiple intents.

[0094] For example, in test case A, the intent of the first test data pair is "turn off the music", and the intent of the second test data pair is "turn up the music volume". According to the logic, the music volume cannot be adjusted after turning off the music, so test case A is not compliant.

[0095] For another example, in test case B, the intent of the first test data pair is "turn down the music volume", and the intent of the second test data pair is "turn off the music". The operation of adjusting the volume first and then turning off the music is logical, so test case B is compliant.

[0096] The test data pair may include input information and an interaction result for the input information, or may include rewritten input information and an interaction result for the input information, so the test case may include rewritten input information.

[0097] Since the rewritten input information may not be logical when the large model is used to rewrite the input information, it is necessary to perform rewriting compliance identification on the rewritten input information. Specifically, when performing rewriting compliance identification on the rewritten input information, it can be determined whether the rewritten input information is logical.

[0098] For example, the input information is "Modify the advertising bid to 20 yuan", and the rewritten input information A is "Set the bid to 20 yuan", which is logical.

[0099] For another example, the rewritten input information B is "Set bid at 20 yuan", which is illogical.

[0100] When the input information is rewritten using a large model, the intent of the rewritten input information may be different from the intent of the input information, resulting in illogicality between the rewritten input information and the interaction result for the input information. Therefore, it is necessary to identify the association compliance between the rewritten input information and the interaction result for the input information. Specifically, when the association compliance between the rewritten input information and the interaction result is identified, it can be determined whether there is an association between the rewritten input information and the interaction result for the input information.

[0101] For example, the input information is "change the ad bid to 20 yuan", and the interactive result for the input information includes calling the ad price modification interface to change the ad bid to 20 yuan. The rewritten input information A is "the bid is set to 20 yuan", which is associated with the interactive result for the input information.

[0102] For another example, if the rewritten input information C is "the bid is set to no limit", then there is no correlation between the rewritten input information C and the interaction result for the input information.

[0103] The embodiments of the present disclosure determine the compliance of test cases from multiple perspectives, ensure the availability of test cases, and improve test quality by performing compliance identification on the execution order of multiple intentions, performing rewriting compliance identification on rewritten input information, and performing association compliance identification on the rewritten input information and interaction results.

[0104] According to an embodiment of the present disclosure, after performing operation S220 as shown in the figure, Figure 2 Before the operation S230 shown, the testing method for the intelligent agent may also include: when the compliance identification result indicates that the test case is non-compliant, performing at least one of the following: adjusting the combination order of multiple test data pairs in the test case; updating at least one test data pair in the test case; and updating the rewritten input information using a large model.

[0105] When it is determined that the execution order of multiple intentions of a test case is not compliant, the combination order of multiple test data pairs can be adjusted so that the execution order of the multiple intentions is compliant.

[0106] For example, in test case A, the intent of the first test data pair is "turn off the music", and the intent of the second test data pair is "turn up the music volume". At this time, the combination order of the first test data pair and the second test data pair can be adjusted to obtain test case C. In test case C, the intent of the first test data pair is "turn up the music volume", and the intent of the second test data pair is "turn off the music". The execution order of multiple intents is compliant, that is, test case C is compliant.

[0107] When it is determined that the execution order of multiple intentions of the test case is not compliant, at least one of the multiple test data pairs may be updated so that the execution order of the multiple intentions of the updated test case is compliant.

[0108] For example, in test case A, the intent of the first test data pair is "turn off the music", and the intent of the second test data pair is "turn up the music volume". At this time, the first test data pair can be updated to obtain test case D. In test case D, the intent of the first test data pair is "turn on the music", and the intent of the second test data pair is "turn up the music volume". The execution order of multiple intents is compliant, that is, test case D is compliant.

[0109] When it is determined that the rewritten input information is non-compliant, the test data pair to which it belongs can be updated. Specifically, the intent set corresponding to the rewritten input information can be determined, and the test data pair can be updated using the input information in the intent set or other rewritten input information.

[0110] If the rewritten input information is determined to be non-compliant, the rewritten input information can also be updated using the large model. Specifically, the non-compliant rewritten input information can be input into the large model, modified to be compliant using the large model, and the modified input information can be used to update the rewritten input information.

[0111] When it is determined that the rewritten input information is not compliant with the interaction result, the test data pair to which it belongs can be updated. Specifically, the intent set corresponding to the rewritten input information can be determined, and the test data pair can be updated using the input information in the intent set or other rewritten input information.

[0112] The embodiments of the present disclosure obtain compliant test cases by updating and adjusting non-compliant test cases and test data pairs, thereby improving the logic of the test cases and further improving the test quality.

[0113] According to the embodiments of the present disclosure, Figure 2 The operation S220 shown combines multiple test data pairs with different intentions to obtain a test case, which may include: when it is determined that the intention is to call an external tool, storing the calling parameters of the target input information in a parameter storage space corresponding to the intention; storing the interaction results of the target input information in a result storage space; generating a task instruction based on the storage address and intention of the result storage space; generating a test data pair based on the task instruction and the target input information; and combining multiple test data pairs with different intentions to obtain a test case.

[0114] The target input information may include input information or rewritten input information. The rewritten input information is obtained by rewriting the input information as described in the above embodiment.

[0115] The parameter storage space may be a storage space for storing call parameters of target input information. Specifically, the call parameters may be input parameters required for calling an external tool or parameters required for obtaining an interaction result.

[0116] The result storage space can be used to store the interaction result of the target input information. Specifically, the interaction result can be the result obtained by calling an external tool for the target input information.

[0117] The task instruction may include prompt information for prompting the tested intelligent agent to complete the task. Specifically, the task instruction may prompt the intelligent agent to realize the intention represented by the target input information, and store the interaction result for the target input information into the result storage space.

[0118] In one embodiment, a task instruction template may be pre-constructed, and after the storage address and intent of the result storage space are obtained, the storage address and intent of the result storage space are filled into the task instruction template to generate a task instruction.

[0119] The test input data in the test data pair may be generated based on the task instruction and the target input information, and the reference result data in the test data pair may also be generated based on the interaction result of the target input information.

[0120] When testing an agent, the test input data can be used to characterize the data input to the agent, and the reference result data can be used to evaluate the test interaction results obtained by the agent based on the test input data.

[0121] After obtaining multiple test data pairs, multiple test data pairs with different intentions can be combined to obtain test cases.

[0122] According to the intent, determine whether the test data needs to call external tools for the corresponding tasks or functions. When it is determined that external tools need to be called, the calling parameters of the target input information are stored in the parameter storage space, and the interaction results of the target input information are stored in the result storage space. And generate task instructions based on the storage address and intent of the result storage space. Make the test case implicitly used to obtain the input parameters and return values ​​for calling external tools. In this way, the corresponding calling parameters and test interaction results can be directly obtained from the parameter storage space and the result storage space, avoiding calling real external tools or rewriting data in the real database, thereby improving the simulation capability of the test scenario while reducing dependence on external tools and improving the test scope and independence of the intelligent agent.

[0123] Figure 6 A schematic diagram of generating test data pairs according to another embodiment of the present disclosure is schematically shown.

[0124] like Figure 6As shown, in embodiment 600, the intent 602 of the target input information can be determined according to the target input information 601. Based on the intent 602 and the storage address 603 of the result storage space, a task instruction 604 is generated, wherein the result storage space is used to store the interaction result. Based on the target input information 601 and the task instruction 604, a test data pair 605 is generated.

[0125] Generating a task instruction 604 based on the intent 602 and the storage address 603 may include: filling the intent 602 and the storage address 603 into a task instruction template to generate the task instruction 604 .

[0126] According to the embodiments of the present disclosure, Figure 2 The operation S230 shown is to test the agent for multiple rounds of interaction using the test case, including: according to the execution order of multiple test data pairs in the test case, input the target input information to be tested in the test case into the agent, so that the agent performs the following operations: determine the intention indicated by the target input information; when it is determined that the intention is to call an external tool, obtain the call parameters required for calling the external tool from the parameter storage space corresponding to the intention; and based on the call parameters, the target input information and the task instruction corresponding to the intention, obtain the test interaction result from the result storage space indicated by the task instruction. The result storage space stores the interaction result matching the target input information.

[0127] The execution order can be used to characterize the arrangement order of multiple test data pairs when the agent is tested in sequence using multiple test data pairs. The target input information currently to be tested can be the input information in the current round of test data pairs or the rewritten input information.

[0128] In response to receiving the target input information, the agent needs to determine the intention indicated by the target input information. Specifically, the intention can be obtained from the target input information or from the task instruction.

[0129] Generally, when testing an agent, it is necessary to configure the agent's test environment, such as the running environment of an external tool, to ensure that the agent can call the external tool to obtain the final interaction result when the intention is to call the external tool. However, since the use scenarios of agents are relatively wide, it may be necessary to call a large number of external tools, which makes it difficult to configure the test environment of the agent.

[0130] Since the same external tool generates the same output parameters for the same input parameters, we can use the calling parameters in the parameter storage space and the interaction results in the result storage space, and the mapping relationship between the calling parameters and the interaction results to simulate the operating logic of the external calling tool, and then get the result of calling the external tool without setting up the operating environment of the external tool.

[0131] When the intelligent agent determines that it needs to call an external tool, it can obtain the calling parameters from the parameter storage space corresponding to the intent, obtain the interaction results that match the calling parameters, target input information, and task instructions corresponding to the intent from the result storage space, and use them as test interaction results to simulate the process of calling the external tool.

[0132] The embodiments of the present disclosure simulate the process of an intelligent agent calling an external tool through stored calling parameters and interaction results, enter an operating environment without setting up external tools, and improve testing efficiency.

[0133] Figure 7 A schematic diagram of generating a test interaction result according to an embodiment of the present disclosure is schematically shown.

[0134] like Figure 7 As shown, in embodiment 700, the agent determines the intention 702 indicated by the target input information 710. After determining that the intention 702 represents the call of the external tool, the call parameter 705 is obtained from the parameter storage space 704. Based on the target input information 701, the task instruction 703 and the call parameter 705, the test interaction result 707 is obtained from the result storage space 706.

[0135] According to an embodiment of the present disclosure, evaluating an intelligent agent includes: performing test evaluation on multiple interaction results and multiple test interaction results in a test case to obtain a test evaluation result.

[0136] Since the test cases include interaction results that can be used as expected results, the interaction results can be used to test and evaluate the test interaction results to obtain the test evaluation results of the intelligent agent.

[0137] In one embodiment, the interaction result and the test interaction result may both be strings in JSON format, so the test evaluation result may be determined based on the difference between the interaction result and the test interaction result.

[0138] Determining the difference between the interaction result and the test interaction result may include: using a large model to determine the difference between the character string of the interaction result and the character string of the test interaction result, wherein the character string may include multiple attribute key-value pairs, the attribute key-value pairs include attribute names and attribute values, and the attribute key-value pairs can be used to characterize call parameters, interaction results, and calling tools, etc.

[0139] When using the big model to determine the difference between the interaction result string and the test interaction result string, you can input a prompt word for determining the difference into the big model. For example, you are a senior agent testing expert. Now there are two strings: the interaction result string and the test interaction result string. Please find the difference between the two and output it. If there is no difference, output "same". The difference between the interaction result string and the test interaction result string is considered from the following aspects: string structure; attribute name in the string; attribute value in the string; number of attributes in the string.

[0140] Based on the output results of the large model, the test evaluation results are obtained. Based on the test evaluation results, the current ability of the agent to perform multiple rounds of interaction is determined. If the test evaluation results meet the test evaluation standards, the agent can be put online for use. If the test evaluation results do not meet the test evaluation standards, more test cases can be constructed using the test method for the agent to determine the problems that exist in the process of the agent performing multiple rounds of interaction, and then the large model, tool call interface, etc. included in the agent are adjusted, thereby improving the overall interaction ability of the agent.

[0141] Figure 8 A block diagram of a testing device for an intelligent agent according to an embodiment of the present disclosure is schematically shown.

[0142] like Figure 8 As shown, the testing device 800 for the intelligent agent includes a data acquisition module 810, a data combination module 820 and an interactive testing module 830.

[0143] The data acquisition module 810 is used to acquire input information and interaction results for the input information to obtain test data pairs.

[0144] The data combination module 820 is used to combine multiple test data pairs with different intentions to obtain test cases.

[0145] The interaction test module 830 is used to perform multiple rounds of interaction tests on the agent using the test case when it is determined that the test case meets the interaction logic standard.

[0146] According to an embodiment of the present disclosure, the data acquisition module includes an input extraction submodule and a data generation submodule.

[0147] The input extraction submodule is used to extract target input information from multiple intention sets with different intentions, wherein the intention set includes input information or rewritten input information, and the rewritten input information is obtained by rewriting the input information using a large model.

[0148] The data generation submodule is used to obtain test data pairs based on target input information and interaction results.

[0149] According to an embodiment of the present disclosure, the data acquisition module further includes a field extraction submodule and a semantic rewriting submodule.

[0150] The field extraction submodule is used to extract key fields from the input information.

[0151] The semantic rewriting submodule is used to semantically rewrite the input information using the large model while retaining key fields to obtain the rewritten input information.

[0152] According to an embodiment of the present disclosure, the apparatus 800 for testing an intelligent agent further includes a compliance identification module and a standard determination module.

[0153] The compliance identification module is used to perform logical compliance identification on test cases and obtain compliance identification results.

[0154] The standard determination module is used to determine whether the test case complies with the interaction logic standard when the compliance identification result indicates that the test case complies.

[0155] According to an embodiment of the present disclosure, the compliance identification module includes at least one of a sequential identification submodule, an overwriting identification submodule, and an association identification submodule.

[0156] The sequence identification submodule is used to identify the compliance of the execution order of multiple intents of the test case.

[0157] The rewriting identification submodule is used to identify the rewriting compliance of the rewritten input information in the test case.

[0158] The association identification submodule is used to identify the association compliance between the rewritten input information and the interaction results.

[0159] According to an embodiment of the present disclosure, the testing device 800 for an intelligent agent further includes a compliance execution module.

[0160] The compliance execution module is used to execute at least one of the following when the compliance identification result indicates that the test case is non-compliant: adjusting the combination order of multiple test data pairs in the test case; updating at least one test data pair in the test case; and updating the rewritten input information using the large model.

[0161] According to an embodiment of the present disclosure, the data combination module includes a parameter storage submodule, an interaction storage submodule, an instruction generation submodule, a data generation submodule and a data combination submodule.

[0162] The parameter storage submodule is used to store the calling parameters of the target input information into the parameter storage space corresponding to the intention when it is determined that the intention is to call an external tool. The target input information includes the input information or the rewritten input information.

[0163] The interaction storage submodule is used to store the interaction results of the target input information in the result storage space.

[0164] The instruction generation submodule is used to generate task instructions based on the storage address and intention of the result storage space.

[0165] The data generation submodule is used to generate test data pairs based on task instructions and target input information.

[0166] The data combination submodule is used to combine multiple test data pairs with different intentions to obtain test cases.

[0167] According to an embodiment of the present disclosure, the interactive test module includes an information input submodule.

[0168] The information input submodule is used to input the target input information to be tested in the test case into the agent according to the execution order of the multiple test data pairs in the test case, so that the agent performs the following operations: determining the intention indicated by the target input information, wherein the target input information includes the input information or the rewritten input information.; in the case where the intention is determined to be to call an external tool, obtaining the call parameters required for calling the external tool from the parameter storage space corresponding to the intention; and based on the call parameters, the target input information and the task instruction corresponding to the intention, obtaining the test interaction result from the result storage space indicated by the task instruction, wherein the result storage space stores the interaction result matching the target input information.

[0169] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0170] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0171] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0172] According to an embodiment of the present disclosure, a computer program product includes a computer program, and the computer program implements the above method when executed by a processor.

[0173] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0174] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0175] A number of components in the device 900 are connected to an input / output (I / O) interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0176] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a test method for an intelligent agent. For example, in some embodiments, the test method for an intelligent agent may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the test method for the intelligent agent described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the test method for the intelligent agent in any other appropriate manner (e.g., by means of firmware).

[0177] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0178] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0179] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0180] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0181] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0182] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0183] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0184] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for testing an intelligent agent, comprising: Acquire input information and an interaction result for the input information to obtain a test data pair; Combining a plurality of test data pairs with different intentions to obtain a test case; as well as When it is determined that the test case meets the interaction logic standard, the test case is used to perform multiple rounds of interaction tests on the agent.

2. The method according to claim 1, wherein: The step of obtaining the input information and the interaction result with respect to the input information to obtain the test data pair includes: Extracting target input information from a plurality of intention sets with different intentions, wherein the intention sets include the input information or rewritten input information, and the rewritten input information is obtained by rewriting the input information using a large model; and The test data pair is obtained based on the target input information and the interaction result.

3. The method according to claim 2, further comprising: Extracting key fields from the input information; as well as While retaining the key fields, the input information is semantically rewritten using the large model to obtain the rewritten input information.

4. The method according to any one of claims 1 to 3, further comprising: Performing logical compliance identification on the test case to obtain a compliance identification result; as well as In a case where the compliance identification result indicates that the test case is compliant, it is determined that the test case complies with the interaction logic standard.

5. The method according to claim 4, wherein: The performing logical compliance identification on the test case to obtain a compliance identification result includes at least one of the following: Perform compliance identification on the execution order of multiple intentions of the test case; Performing rewriting compliance identification on the rewritten input information in the test case; as well as The rewritten input information and the interaction result are associated with each other for compliance identification.

6. The method according to claim 4 or 5, further comprising: If the compliance identification result indicates that the test case is non-compliant, at least one of the following is performed: Adjusting the combination order of the plurality of test data pairs in the test case; updating at least one of the test data pairs in the test case; The rewritten input information is updated using the large model.

7. The method according to any one of claims 1 to 6, wherein: Combining a plurality of test data pairs with different intentions to obtain a test case includes: In the case where it is determined that the intention is to call an external tool, storing the calling parameters of the target input information in a parameter storage space corresponding to the intention, wherein the target input information includes the input information or the rewritten input information; Storing the interaction result of the target input information in the result storage space; generating a task instruction based on the storage address of the result storage space and the intention; generating the test data pair based on the task instruction and the target input information; and The test case is obtained by combining a plurality of test data pairs with different intentions.

8. The method according to any one of claims 1 to 6, wherein: The testing of performing multiple rounds of interactions on the agent using the test case includes: According to the execution order of the multiple test data pairs in the test case, the target input information to be tested in the test case is input into the agent, so that the agent performs the following operations: determining the intention indicated by the target input information, wherein the target input information includes the input information or the rewritten input information; In the case where it is determined that the intention is to call an external tool, acquiring a calling parameter required for calling the external tool from a parameter storage space corresponding to the intention; and Based on the calling parameters, the target input information and the task instruction corresponding to the intention, the test interaction result is obtained from the result storage space indicated by the task instruction, wherein the result storage space stores the interaction result matching the target input information.

9. A testing device for an intelligent agent, comprising: A data acquisition module, used to acquire input information and interaction results for the input information to obtain a test data pair; A data combination module, used for combining a plurality of test data pairs with different intentions to obtain a test case; as well as The interactive testing module is used to perform multiple rounds of interactive testing on the intelligent agent using the test case when it is determined that the test case meets the interactive logic standard.

10. The device according to claim 9, wherein: The data acquisition module comprises: an input extraction submodule, configured to extract target input information from a plurality of intent sets having different intents, wherein the intent sets include the input information or rewritten input information, wherein the rewritten input information is obtained by rewriting the input information using a large model; and The data generation submodule is used to obtain the test data pair based on the target input information and the interaction result.

11. The apparatus according to claim 10, further comprising: A field extraction submodule, used to extract key fields from the input information; as well as The semantic rewriting submodule is used to semantically rewrite the input information using the large model while retaining the key fields to obtain the rewritten input information.

12. The device according to any one of claims 9 to 11, further comprising: A compliance identification module, used to perform logical compliance identification on the test case to obtain a compliance identification result; as well as The standard determination module is used to determine that the test case complies with the interactive logic standard when the compliance identification result indicates that the test case complies.

13. The device according to claim 12, wherein: The compliance identification module includes at least one of the following: A sequence identification submodule, used for identifying the compliance of the execution sequence of multiple intentions of the test case; A rewriting identification submodule, used for performing rewriting compliance identification on the rewritten input information in the test case; as well as The association identification submodule is used to perform association compliance identification on the rewritten input information and the interaction result.

14. The device according to claim 12 or 13, further comprising: A compliance execution module is configured to execute at least one of the following when the compliance identification result indicates that the test case is non-compliant: Adjusting the combination order of the plurality of test data pairs in the test case; updating at least one of the test data pairs in the test case; The rewritten input information is updated using the large model.

15. The method according to any one of claims 9 to 14, wherein: The data combination module comprises: a parameter storage submodule, for storing the calling parameters of the target input information into a parameter storage space corresponding to the intention when it is determined that the intention is to call an external tool, wherein the target input information includes the input information or the rewritten input information; An interactive storage submodule, used for storing the interactive result of the target input information into a result storage space; An instruction generation submodule, used for generating a task instruction based on the storage address of the result storage space and the intention; a data generation submodule, configured to generate the test data pair based on the task instruction and the target input information; and The data combination submodule is used to combine a plurality of test data pairs with different intentions to obtain the test case.

16. The method according to any one of claims 9 to 14, wherein: The interactive test module includes: The information input submodule is used to input the target input information to be tested in the test case into the agent according to the execution order of the multiple test data pairs in the test case, so that the agent performs the following operations: determining the intention indicated by the target input information, wherein the target input information includes the input information or the rewritten input information; In the case where it is determined that the intention is to call an external tool, acquiring a calling parameter required for calling the external tool from a parameter storage space corresponding to the intention; and Based on the calling parameters, the target input information and the task instruction corresponding to the intention, the test interaction result is obtained from the result storage space indicated by the task instruction, wherein the result storage space stores the interaction result matching the target input information.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images