Intelligent agent testing method and device, readable medium, electronic equipment and program product
By calculating the expected reward value of large language model software products in different states, the action with the maximum expected reward value is preferred, which solves the problem of incomplete coverage of test cases and improves test efficiency and coverage.
Patent Information
- Application Number
- CN202411897940.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-03-18
AI Technical Summary
During the testing of large language model software products, due to the diversity and uncertainty of inputs and outputs, test cases are difficult to cover all functions or requirements, resulting in low testing efficiency and coverage.
By determining the current object state and multiple candidate actions of the agent to be tested, calculating the expected object state and expected reward value after each action, the action with the maximum expected reward value is preferred to improve testing efficiency and coverage.
By prioritizing the execution of actions with the largest expected reward value, actions that are not executed during the historical testing can be improved to improve test coverage and efficiency, and automatic testing of software products connected to machine learning models can be realized.
Smart Images
Figure CN119645877A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the fields of large language models, large models, intelligent agents, and artificial intelligence technology, and in particular, to an intelligent agent testing method, device, readable medium, electronic device, and program product. Background Art
[0002] With the continuous development of machine learning models, an increasing number of software products are integrating machine learning models such as Large Language Models (LLMs). During the testing of these software products, due to the rich, diverse, and uncertain inputs and outputs of machine learning models, test cases struggle to cover all functions or requirements, thus impacting test efficiency and coverage. Summary of the Invention
[0003] This summary is provided to briefly introduce concepts that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.
[0004] In a first aspect, the present disclosure provides an intelligent agent testing method, the intelligent agent testing method comprising: Determining a current object state and a plurality of candidate actions of an agent to be tested, wherein the agent to be tested is a software product connected to a first machine learning model, the current object state is the current state of a target object interacting with the agent to be tested, and the candidate actions are actions that the agent to be tested can perform under the current object state; For each candidate action, determining an expected object state after the agent to be tested performs the candidate action, and determining an expected reward value obtained by the agent to be tested after performing the candidate action based on the expected object state and the target object state of the agent to be tested, wherein the expected reward value is used to represent the minimum difference between the expected object state and the target object state, and the target object state includes the current object state and the historical object state of the agent to be tested; The candidate action with the largest expected reward value is preferentially determined as the target action, and the intelligent agent to be tested is controlled to execute the target action.
[0005] In a second aspect, the present disclosure provides an intelligent agent testing device, the intelligent agent testing device comprising: a first determination module, configured to determine a current object state of an agent to be tested and a plurality of candidate actions, wherein the agent to be tested is a software product connected to a first machine learning model, the current object state is the current state of a target object interacting with the agent to be tested, and the candidate actions are actions that the agent to be tested can perform under the current object state; a second determination module, configured to determine, for each candidate action, an expected object state after the agent to be tested performs the candidate action, and determine, based on the expected object state and the target object state of the agent to be tested, an expected reward value obtained after the agent to be tested performs the candidate action, wherein the expected reward value is used to represent a minimum difference value between the expected object state and the target object state, and the target object state includes the current object state and the historical object state of the agent to be tested; The control module is used to preferentially determine the candidate action with the largest expected reward value as the target action, and control the intelligent agent to be tested to execute the target action.
[0006] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when executed by a processing device.
[0007] In a fourth aspect, the present disclosure provides an electronic device, comprising: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the method in the first aspect.
[0008] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.
[0009] Through the above technical solution, for each candidate action that the tested agent can perform under the current object state, the expected object state after the tested agent executes the candidate action can be determined. Based on the expected object state and the tested agent's existing object state (i.e., the current object state and the historical object state), the expected reward value obtained by the tested agent after executing the candidate action can be determined. Finally, the tested agent is controlled to prioritize the candidate action with the highest expected reward value. Using this method, because the expected reward value is the minimum difference between the tested agent's expected object state and the existing object state, prioritizing the action with the highest expected reward value is equivalent to prioritizing the action with the largest difference from the existing object state. This allows for the prioritization of actions that were not executed during historical testing. In other words, reinforcement learning can be used to automatically test software products integrated with machine learning models, thereby improving test efficiency and coverage.
[0010] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 is a flow chart illustrating an intelligent agent testing method according to an exemplary embodiment of the present disclosure; Figure 2 is a page diagram illustrating a method for running a test according to an exemplary embodiment of the present disclosure; Figure 3 is a process diagram of an intelligent agent testing method according to an exemplary embodiment of the present disclosure; Figure 4 is a schematic diagram illustrating an execution process of an intelligent agent testing method according to an exemplary embodiment of the present disclosure; Figure 5 is a structural block diagram of an intelligent agent testing device according to an exemplary embodiment of the present disclosure; Figure 6 The figure is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0012] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.
[0013] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0014] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.
[0015] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0016] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".
[0017] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.
[0018] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0019] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.
[0020] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.
[0021] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.
[0022] At the same time, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.
[0023] There are three common approaches to testing software products developed based on large language models: 1. Crowdsourced testing: This involves crowdsourcing testing to different testers, who can then simulate a wide variety of complex usage scenarios. However, while crowdsourced testing can simulate diverse usage scenarios, its results lack controllability and consistency, making quality assurance difficult. Furthermore, managing and coordinating testers presents significant challenges, particularly in ensuring consistent and timely feedback. Consequently, this testing approach is inefficient.
[0024] 2. Static Analysis: While the runtime of large language models is unpredictable, the code itself remains deterministic. Therefore, related technologies reduce runtime uncertainty by deepening software analysis. While static analysis can help identify potential issues in the code, it cannot fully predict the software's actual runtime behavior, particularly the dynamic issues that may arise when large language models process complex data. Furthermore, static analysis may not capture all the complexities of model training and data processing. Consequently, this approach offers limited test coverage.
[0025] 3. Main process analysis based on product traffic flow: Leverage product design documentation to map out the expected user traffic flow for the product. Simulate and recreate these key scenarios as closely as possible to ensure the software's main flow meets expectations. While this approach ensures that the software's critical path meets design expectations, it overlooks edge cases and unusual processes in actual user use. Furthermore, as product features evolve and user behavior changes, the original main process analysis can quickly become outdated, requiring frequent updates to keep pace with product development. Consequently, this approach offers low testing efficiency and coverage.
[0026] In addition, due to the generative nature of software products developed based on large language models, there is increased uncertainty in joint debugging, testing, and regression during the product development process. This is mainly reflected in the following aspects: 1. Uncertain output variation: Because large language models can produce rich and varied outputs based on given inputs, even small model updates or input changes can lead to significant differences in output. This uncertainty complicates the prediction and control of model behavior and increases the difficulty of regression testing.
[0027] 2. Test coverage: In traditional software development, test coverage is a key indicator of test quality. However, in large language model products, due to the high diversity of inputs and outputs, how to effectively select test cases to cover the most critical usage scenarios and areas most likely to cause problems becomes a pressing issue.
[0028] 3. Detecting Logical and Reasoning Errors: Large language models may produce errors when processing complex logic or performing reasoning. These errors are not as easy to pinpoint and fix as code defects. Testing needs to verify that the model can accurately understand instructions and provide reasonable output in various situations. This requires designing test cases based on highly specialized knowledge and complex test cases.
[0029] 4. Long-term stability and reliability: Maintaining the stability and reliability of large language model products is an ongoing challenge as models are continuously updated over time. After each model iteration, comprehensive regression testing is required to confirm that the update does not introduce new issues or negatively impact existing functionality.
[0030] It can be seen that the testing process of software products developed based on large language models greatly increases the testing complexity compared with the testing process of traditional software.
[0031] In view of this, the present disclosure provides an intelligent agent testing method, device, readable medium, electronic device and program product to improve the testing efficiency and test coverage of software products connected to machine learning models.
[0032] The following further explains the embodiments of the present disclosure with reference to the accompanying drawings.
[0033] Figure 1 This is a flow chart of an agent testing method according to an exemplary embodiment of the present disclosure, with reference to Figure 1 , the agent testing method may include the following steps: S101: Determine the current object state and multiple candidate actions of the agent to be tested, wherein the agent to be tested is a software product connected to the first machine learning model, the current object state is the current state of the target object interacting with the agent to be tested, and the candidate actions are actions that the agent to be tested can perform under the current object state.
[0034] For example, the first machine learning model can be a large language model, or other neural network models, deep learning models, etc., and this disclosure does not limit this. The intelligent agent to be tested can be a software product that is connected to a large language model, such as an intelligent dialogue product that is connected to a large language model, and the intelligent dialogue product can interact with the user based on the large language model. Actions can be understood as dialogues, clicks, slides, and other actions on the software product. The specific actions can be determined based on the actual business scenario. For example, in an intelligent dialogue product, entering different query contents can be regarded as different actions, and this disclosure does not limit this.
[0035] For example, the target object can be understood as a virtual user used for software testing. It should be noted that an object state of a target object can include multiple sub-states, each of which can be represented as a sequence of key-value pairs. For example, an object state can be represented by {"timestamp": xx, "input": yes, "first answer": no, "number of conversation turns": 3, ...}. The specific details can be determined based on actual business scenarios and are not limited in this disclosure.
[0036] Therefore, a state of the target object can also be understood as a test case. For example, taking the intelligent dialogue product in the advertising scenario as an example, there may be states such as "the user did not ask", "the user asked for diagnostic suggestions from advertisement A, but did not get a reply", "the user asked for diagnostic suggestions from advertisement A and got an incorrect reply", "the user asked for diagnostic suggestions from advertisement A and got a correct reply", etc. During the testing process, it is necessary to cover these test cases as much as possible. In the embodiment of the present disclosure, the state space can be used to define the set of test cases that can be covered by the software product connected to the machine learning model. For example, the state space X={S} is defined, where S represents all states that X can reach. For business scenarios of software products connected to machine learning models, the state space is usually not finite, so it is hoped to produce and cover these states as much as possible.
[0037] Under different object states, the actions that the tested agent can perform may be different. Therefore, in order to realize automatic testing of the tested agent in the embodiment of the present disclosure, the current object state of the tested agent and the actions that can be performed under the current object state are first determined.
[0038] S102: For each candidate action, determine the expected object state after the tested agent executes the candidate action, and determine the expected reward value obtained by the tested agent after executing the candidate action based on the expected object state and the target object state of the tested agent.
[0039] The expected reward value represents the minimum difference between the expected object state and the target object state. The target object state includes the current object state and the historical object states of the tested agent. In other words, the expected reward value is derived based on the expected object state and all previous object states of the tested agent. This allows the tested agent to determine subsequent actions based on all previously executed actions, thereby reducing repeated actions and improving test coverage.
[0040] S103: Prioritize the candidate action with the largest expected reward value as the target action, and control the tested intelligent agent to execute the target action.
[0041] For example, the above process can be implemented based on reinforcement learning. For example, a greedy algorithm (ε-greedy) can be used to determine the target action from candidate actions. This means that the action with the highest expected reward value is selected most of the time to explore the maximum reward, but there is a small probability (ε) that any possible action from the candidate actions will be randomly selected to explore the environment. Of course, other methods similar to the greedy algorithm can also be used to prioritize the action with the highest expected reward value, and this disclosure is not limited to this.
[0042] For example, after the intelligent agent to be tested performs the target action, the intelligent agent to be tested can observe the new environment and new object state, and then repeat steps S101-S103 until the preset test completion conditions are met, such as the test coverage reaches a preset threshold, etc. This disclosure does not limit this.
[0043] Take the testing process of the above-mentioned intelligent dialogue product as an example. Figure 2 As shown, automatic questions and answers can be performed on the dialogue robot page without the need for human intervention to enter content in the input box, thereby enabling automatic testing of software products that are connected to machine learning models.
[0044] Using the above method, since the expected reward value is the minimum difference between the expected object state of the intelligent agent to be tested and the existing object state, the action with the largest expected reward value is executed first, which is equivalent to giving priority to the action with a greater difference from the existing object state. Therefore, the actions that have not been executed in the historical test process can be executed first. In other words, software products connected to machine learning models can be automatically tested based on reinforcement learning, thereby improving test efficiency and test coverage.
[0045] In a possible manner, determining multiple candidate actions of the intelligent agent to be tested may include: determining executable actions of the intelligent agent to be tested, the executable actions being obtained by simulating user behavior of the intelligent agent to be tested based on a third machine learning model; and screening multiple candidate actions from the executable actions according to the environment in which the intelligent agent to be tested is located and the current object state.
[0046] It should be noted that the third machine learning model can be a large language model, or other neural network models, deep learning models, etc., and this disclosure does not limit this.
[0047] For example, the user behavior of the user in the tested agent can be simulated based on the third machine learning model, such as Figure 3 The model A shown above constructs a prompt word a based on user status, test documents and other information before starting the test. The prompt word a is used to guide model A to simulate the user's behavior in the intelligent agent to be tested. Taking the intelligent dialogue product in the advertising scenario as an example, the prompt word "You are a user of an advertising platform. You encounter problems such as 'xxx' and need to ask questions to customer service. The historical dialogue is as follows 'xxx'. Please continue the dialogue" can be determined according to the actual business scenario, and this disclosure does not impose any restrictions on this.
[0048] Then, we input the prompt word a into model A to obtain the next action output by model A. After the agent under test executes the next action, we continue to construct a new prompt word and input it into model A to determine the next action. This process is repeated until we have executed as many possible actions as possible for the agent under test in different environments and user states as possible to obtain an action set. Furthermore, we can obtain the executed actions during subsequent testing to maintain and update the action set.
[0049] Continue to refer to Figure 3 During the testing of the agent to be tested, multiple actions that can be executed in the environment can be screened from the action set based on the environment in which the agent to be tested is located. Then, based on the current object state S1, multiple candidate actions that can be executed in the state can be screened from the multiple actions. Of course, it is also possible to first screen based on the current object state and then based on the environment, and this disclosure does not limit this.
[0050] For example, the environment in which the intelligent agent to be tested is located can be the current page environment. Continuing with the example of the conversation robot in the advertising scenario, for example, the current conversation page includes control 1, then the actions that can be performed by the current conversation page include triggering actions on control 1, which can be specifically determined based on the actual application scenario, and this disclosure does not impose any restrictions on this.
[0051] It is worth noting that, in the process of constructing the action set, the corresponding state can be recorded to construct the state space, and in the test process, the state space can be updated based on the new object state. The specific details can be determined according to the actual application scenario, and this disclosure does not limit this.
[0052] In a possible manner, determining the expected object state after the tested intelligent agent executes the candidate action includes: obtaining historical interaction information of the target object and the tested intelligent agent and a test document for the tested intelligent agent; determining the test phase information of the tested intelligent agent based on the current object state, historical interaction information, test document and a second machine learning model, the second machine learning model being used to determine the test phase information of the tested intelligent agent based on input content; and determining the expected object state based on the test phase information, the current object state and historical interaction information.
[0053] It should be noted that the second machine learning model can be a large language model, or other neural network models, deep learning models, etc., and this disclosure does not limit this.
[0054] For example, the test phase of the agent to be tested can be determined based on the second machine learning model, such as Figure 3 Model B, as shown, can first construct a prompt word b based on the current object state, historical interaction information, and test documents. Prompt word b is used to guide model B in determining the test phase of the agent to be tested. Prompt word b is then input into model B, resulting in test phase information output by model B. The test phase information, current object state, and historical interaction information are then combined to obtain the expected object state. For example, the expected object state can be obtained by directly concatenating the test phase information, current object state, and historical interaction information. The specific determination can be made based on actual needs and is not limited in this disclosure.
[0055] It should be noted that a second machine learning model can also determine the expected object state based on the input content. This can be determined by training the machine learning model as needed, and this disclosure does not impose any restrictions on this. By combining the current object state, historical interaction information, test documents, and other content to predict the expected object state, the accuracy of the predicted expected object state can be improved, thereby improving the accuracy of the expected reward value.
[0056] In a possible manner, the expected reward value obtained by the tested intelligent agent after executing the candidate action is determined based on the expected object state and the target object state of the tested intelligent agent, including: obtaining at least one state difference value based on at least a state vector corresponding to the expected object state and a state vector corresponding to each target object state; and determining the minimum difference value among the at least one state difference value as the expected reward value obtained by the tested intelligent agent after executing the candidate action.
[0057] For example, the state difference between two states can be determined based on the difference in state vectors between the two states. Assuming the historical object state is S0, the current object state is S1, and the expected object state is S2, the difference value 1 between the state vectors S0 and S2, and the difference value 2 between the state vectors S1 and S2 can be calculated, respectively. The minimum value between difference value 1 and difference value 2 is used as the expected reward value obtained after the tested intelligent agent executes the candidate action. The difference value can be the cosine distance between the two vectors, which can be determined specifically based on needs and is not limited by this disclosure.
[0058] In a possible manner, at least one state difference value is obtained based on at least the state vector corresponding to the expected object state and the state vector corresponding to each target object state, including: determining a first state vector corresponding to the expected object state, a first description content in natural language form corresponding to the expected object state, a second state vector corresponding to each target object state, and a second description content in natural language form corresponding to each target object state; for each target object state, determining a first difference value between the first state vector and the second state vector corresponding to the target object state, and determining a second difference value between the first description content and the second description content corresponding to the target object state; and determining a state difference value between the expected object state and the target object state based on the first difference value and the second difference value.
[0059] For example, the state difference value can be determined by combining the state vector and the description content in natural language between the two states to further improve the calculation accuracy of the state difference value. For example, the state difference value between the two states can be determined by the following calculation formula:
[0060] Wherein, S and S' respectively represent the two states for which the state difference value is to be calculated, D represents the state difference value between state S and state S', T represents the state vector of the state, M represents the natural language description content of the state, d represents the first difference value between the two states, for example, d is the cosine distance between the two vectors, G represents the second difference value between the two states, for example, G can be the text difference degree of the description content corresponding to the two states, which can be determined according to the needs, and this disclosure does not limit this. w1 and w2 are preset weights, which can be set according to the needs, and this disclosure does not limit this.
[0061] It is worth noting that the state can be described in natural language through a large language model, and then the difference values between the natural language description contents corresponding to the two states can be compared based on the large language model. The specific settings can be made according to needs, and this disclosure does not impose any restrictions on this.
[0062] For example, the state vector may be described as a normalized vector, that is, T may also represent the normalized state vector of the state, and the first difference value may be determined based on the following calculation formula:
[0063] Among them, A represents the normalized vector description set of state S, B represents the normalized vector description set of state S', a represents the normalized vector description of a sub-state in set A, and b represents the normalized vector description of a sub-state in set B.
[0064] After obtaining multiple state difference values between the expected object state and the target object states, the minimum difference value is used as the expected reward value.
[0065] For example, the above process of determining the expected reward value can be achieved by Figure 3 The illustrated implementation of Model C involves constructing a prompt word c that instructs Model C to calculate an expected reward value. For example, the prompt word "determine the state difference value between the expected object state S and each target object state S'" can be constructed. The specific implementation can be determined based on actual business scenarios and is not limited in this disclosure. Based on the prompt word c and Model C, multiple state difference values between the expected object state and each target object state are then determined, and the minimum difference value is used as the expected reward value.
[0066] It is worth mentioning that Figure 3 Model A, Model B, and Model C in the example can be a large language model, or other neural network models, deep learning models, machine learning models, etc., and the present disclosure does not limit this. By constructing different training samples, multiple models are trained separately to obtain Model A, Model B, and Model C with corresponding functions. Of course, Model A, Model B, and Model C can also be integrated into one model, and then the corresponding model results can be output according to different input prompt words. The specific requirements can be determined, and the present disclosure does not limit this.
[0067] In a possible manner, the expected reward value obtained by the intelligent agent to be tested after executing the candidate action is determined based on the expected object state and the target object state of the intelligent agent to be tested, including: for the first object state and the second object state in the target object state, determining the third difference value between the first object state and the second object state, and when the third difference value is less than a preset difference threshold, merging the first object state and the second object state to obtain a merged object state; based on the merged object state and the expected object state, determining the expected reward value obtained by the intelligent agent to be tested after executing the candidate action.
[0068] For example, to improve the robustness and efficiency of the test, the target object states can be merged to simplify the calculation process for determining the expected reward value. Among the target object states, states with a state difference value less than a preset difference threshold are merged. The process for calculating the state difference value is the same as described above and will not be repeated here. For example, if there were originally 10 target object states, they would be merged into 4 target object states, greatly simplifying the subsequent calculation process for determining the expected reward value.
[0069] It is worth noting that the states in the state space can also be merged, thereby simplifying the state space and improving the test efficiency of the subsequent test process.
[0070] In a possible manner, after controlling the intelligent agent to be tested to perform the target action, the testing method also includes: determining the actual reward value after performing the target action; determining the target reward value corresponding to the target action based on the actual reward value and expected reward value corresponding to the target action, and the target reward value is used to indicate the expected reward value of the intelligent agent to be tested to determine a new candidate action in a new testing process.
[0071] For example, Figure 3 As shown, after executing the target action, the actual reward value after the target action is determined, and then the target reward value corresponding to the target action is determined based on the actual reward value and the expected reward value. The present disclosure does not limit the calculation process of the target reward value. For example, you can refer to the algorithm for updating the reward value in the Q table in the Q-Learning algorithm.
[0072] In possible ways, such as Figure 4 As shown, each state in the state space can be represented by (S t , A t , R t , S t+1 ) is recorded, indicating that the state S t Execute action A t The expected reward value is R t (ie state S t and state S t+1 The state difference between the two), and the state difference between the two when the agent performs action A t The new state after is S t+1 In addition, the next time according to the state S t Determine to execute action A t After that, you can observe the execution of action A t The new state after that, and recalculate the new expected reward value R t+1 , then update the state space, and so on, which will not be described in detail in this disclosure.
[0073] In the subsequent testing process, the expected reward value can be determined by relying on the above-mentioned method of calculating the minimum difference value, or the expected reward value recorded in the state space can be directly obtained. The specific method can be determined according to needs. For example, the expected reward value can be determined by calculation at the beginning of the test, and the state space can be continuously expanded to cover all functions or requirements of the software product as much as possible. In the later stage of the test or subsequent regression testing, the expected reward value recorded in the state space is obtained to determine whether the update of the model or software product introduces new problems or affects the functions of the software product. The present disclosure does not impose any restrictions on this.
[0074] Notably, the testing solution provided by the disclosed embodiments can continuously expand the state space based on reinforcement learning to cover as many requirements or functions as possible for a software product, facilitating verification of the model's ability to accurately understand instructions and provide reasonable outputs in various situations. Furthermore, actions with high reward values are prioritized, thereby increasing the test coverage of the software product, thereby improving its robustness and ensuring its stability and reliability.
[0075] Based on the same concept, the present disclosure provides an intelligent agent testing device, such as Figure 5 As shown, the intelligent agent testing device 500 includes: A first determination module 501 is configured to determine a current object state of an agent to be tested and a plurality of candidate actions, wherein the agent to be tested is a software product connected to a first machine learning model, the current object state is the current state of a target object interacting with the agent to be tested, and the candidate actions are actions that the agent to be tested can perform under the current object state; A second determination module 502 is configured to determine, for each candidate action, an expected object state after the agent to be tested performs the candidate action, and determine an expected reward value obtained by the agent to be tested after performing the candidate action based on the expected object state and the target object state of the agent to be tested, wherein the expected reward value is used to represent a minimum difference between the expected object state and the target object state, and the target object state includes the current object state and the historical object state of the agent to be tested; The control module 503 is configured to preferentially determine the candidate action with the largest expected reward value as the target action, and control the intelligent agent to be tested to execute the target action.
[0076] Optionally, the second determining module 502 is configured to: Acquire historical interaction information between the target object and the agent to be tested and a test document for the agent to be tested; Determining the test phase information of the agent to be tested based on the current object state, the historical interaction information, the test document, and a second machine learning model, wherein the second machine learning model is used to determine the test phase information of the agent to be tested based on input content; The expected object state is determined according to the test phase information, the current object state and the historical interaction information.
[0077] Optionally, the second determining module 502 is configured to: Obtaining at least one state difference value based on at least a state vector corresponding to the expected object state and a state vector corresponding to each of the target object states; The minimum difference value among the at least one state difference value is determined as the expected reward value obtained by the tested intelligent agent after executing the candidate action.
[0078] Optionally, the second determining module 502 is configured to: Determining a first state vector corresponding to the expected object state, a first description content in natural language corresponding to the expected object state, a second state vector corresponding to each of the target object states, and a second description content in natural language corresponding to each of the target object states; For each target object state, determine a first difference value between the first state vector and the second state vector corresponding to the target object state, and determine a second difference value between the first description content and the second description content corresponding to the target object state; and determine a state difference value between the expected object state and the target object state based on the first difference value and the second difference value.
[0079] Optionally, the intelligent agent testing device 500 further includes: A third determination module is used to determine the actual reward value after performing the target action; The fourth determination module is used to determine the target reward value corresponding to the target action based on the actual reward value and the expected reward value corresponding to the target action. The target reward value is used to indicate the expected reward value of the new candidate action determined by the tested intelligent agent in the new test process.
[0080] Optionally, the second determining module 502 is configured to: determining, for a first object state and a second object state in the target object state, a third difference value between the first object state and the second object state, and merging the first object state and the second object state to obtain a merged object state when the third difference value is less than a preset difference threshold; An expected reward value obtained by the tested agent after executing the candidate action is determined according to the merged object state and the expected object state.
[0081] Optionally, the first determining module 501 is configured to: Determining an executable action of the agent to be tested, where the executable action is obtained by simulating user behavior of the agent to be tested based on a third machine learning model; The plurality of candidate actions are screened out from the executable actions according to the environment in which the agent to be tested is located and the current object state.
[0082] Based on the same concept, an embodiment of the present disclosure also provides a computer-readable medium on which a computer program is stored. When the program is executed by a processing device, the steps of the above-mentioned intelligent agent testing method are implemented.
[0083] Based on the same concept, an embodiment of the present disclosure further provides an electronic device, which may include: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the above-mentioned intelligent agent testing method.
[0084] Based on the same concept, an embodiment of the present disclosure also provides a computer program product, including a computer program, which implements the steps of the above-mentioned intelligent agent testing method when executed by a processor.
[0085] Reference below Figure 6 , which shows a schematic structural diagram of an electronic device 600 suitable for implementing an embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.
[0086] like Figure 6As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or programs loaded from a storage device 608 into a random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.
[0087] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.
[0088] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
[0089] It should be noted that the computer-readable medium described above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.
[0090] In some embodiments, communications may be conducted using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.
[0091] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0092] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: determines the current object state and multiple candidate actions of the agent to be tested, wherein the agent to be tested is a software product connected to a first machine learning model, the current object state is the current state of the target object interacting with the agent to be tested, and the candidate actions are actions that the agent to be tested can perform under the current object state; for each candidate action, determines the expected object state after the agent to be tested performs the candidate action, and determines the expected reward value obtained by the agent to be tested after performing the candidate action based on the expected object state and the target object state of the agent to be tested, wherein the expected reward value is used to represent the minimum difference value between the expected object state and the target object state, and the target object state includes the current object state and the historical object state of the agent to be tested; preferentially determines the candidate action with the largest expected reward value as the target action, and controls the agent to be tested to perform the target action.
[0093] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0094] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0095] The modules described in the embodiments of the present disclosure may be implemented in software or hardware, wherein the name of a module does not necessarily limit the module itself.
[0096] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0097] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0098] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the scope of the above disclosure. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.
[0099] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0100] Although the subject matter has been described using language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the above-described embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method and will not be elaborated upon here.
Claims
1. An agent testing method, characterized in that: The agent testing method comprises: Determine a current object state and a plurality of candidate actions of an intelligent agent to be tested, wherein the intelligent agent to be tested is a software product connected to a first machine learning model, the current object state is a current state of a target object interacting with the intelligent agent to be tested, and the candidate actions are actions that the intelligent agent to be tested can perform under the current object state; For each of the candidate actions, determine the expected object state after the tested intelligent agent performs the candidate action, and determine the expected reward value obtained by the tested intelligent agent after performing the candidate action according to the expected object state and the target object state of the tested intelligent agent, wherein the expected reward value is used to represent the minimum difference value between the expected object state and the target object state, and the target object state includes the current object state and the historical object state of the tested intelligent agent; The candidate action with the largest expected reward value is preferentially determined as the target action, and the intelligent agent to be tested is controlled to execute the target action.
2. The agent testing method according to claim 1, characterized in that: The step of determining the expected object state after the tested agent executes the candidate action comprises: Acquire historical interaction information between the target object and the subject to be tested and a test document for the subject to be tested; Determine the test phase information of the subject to be tested based on the current object state, the historical interaction information, the test document and the second machine learning model, wherein the second machine learning model is used to determine the test phase information of the subject to be tested based on input content; The expected object state is determined according to the test phase information, the current object state and the historical interaction information.
3. The agent testing method according to claim 1, characterized in that: The step of determining the expected reward value obtained by the subject to be tested after executing the candidate action according to the expected object state and the target object state of the subject to be tested comprises: Obtaining at least one state difference value based on at least a state vector corresponding to the expected object state and a state vector corresponding to each of the target object states; The minimum difference value among the at least one state difference value is determined as the expected reward value obtained after the tested intelligent agent executes the candidate action.
4. The agent testing method according to claim 3, characterized in that: The obtaining of at least one state difference value according to at least the state vector corresponding to the expected object state and the state vector corresponding to each of the target object states comprises: Determine a first state vector corresponding to the expected object state, a first description content in natural language corresponding to the expected object state, a second state vector corresponding to each of the target object states, and a second description content in natural language corresponding to each of the target object states; For each target object state, determine a first difference value between the first state vector and the second state vector corresponding to the target object state, and determine a second difference value between the first description content and the second description content corresponding to the target object state; and determine a state difference value between the expected object state and the target object state based on the first difference value and the second difference value.
5. The agent testing method according to any one of claims 1 to 4, characterized in that: After controlling the tested intelligent agent to perform the target action, the intelligent agent testing method further includes: Determining an actual reward value after performing the target action; According to the actual reward value and the expected reward value corresponding to the target action, the target reward value corresponding to the target action is determined, and the target reward value is used to indicate the expected reward value of the new candidate action determined by the tested intelligent agent in a new test process.
6. The agent testing method according to any one of claims 1 to 4, characterized in that: The step of determining the expected reward value obtained by the subject to be tested after executing the candidate action according to the expected object state and the target object state of the subject to be tested comprises: For a first object state and a second object state in the target object state, determining a third difference value between the first object state and the second object state, and when the third difference value is less than a preset difference threshold, merging the first object state and the second object state to obtain a merged object state; According to the combined object state and the expected object state, an expected reward value obtained by the tested intelligent agent after executing the candidate action is determined.
7. The agent testing method according to any one of claims 1 to 4, characterized in that: The step of determining a plurality of candidate actions of the intelligent agent to be tested comprises: Determine an executable action of the intelligent agent to be tested, where the executable action is obtained by simulating user behavior of the intelligent agent to be tested based on a third machine learning model; The plurality of candidate actions are selected from the executable actions according to the environment in which the intelligent agent to be tested is located and the current object state.
8. An intelligent agent testing device, characterized in that: The intelligent agent testing device comprises: A first determination module is used to determine a current object state and a plurality of candidate actions of an intelligent subject to be tested, wherein the intelligent subject to be tested is a software product connected to a first machine learning model, the current object state is a current state of a target object interacting with the intelligent subject to be tested, and the candidate actions are actions that the intelligent subject to be tested can perform under the current object state; A second determination module is used to determine, for each of the candidate actions, the expected object state after the tested intelligent agent performs the candidate action, and determine the expected reward value obtained after the tested intelligent agent performs the candidate action according to the expected object state and the target object state of the tested intelligent agent, wherein the expected reward value is used to represent the minimum difference value between the expected object state and the target object state, and the target object state includes the current object state and the historical object state of the tested intelligent agent; The control module is used to preferentially determine the candidate action with the largest expected reward value as the target action, and control the intelligent agent to be tested to execute the target action.
9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 7 are implemented.
10. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 7.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.