Agent test case multi-dimensional coverage evaluation method based on pca dimension reduction

Abstract

The application discloses an agent test case multi-dimension coverage evaluation method based on PCA dimension reduction, belongs to the field of reinforcement learning and agent testing, and mainly comprises the following steps: constructing an agent test case evaluation index, including 2D input space coverage, 3D input space coverage, 2D state space coverage and 3D state space coverage; selecting a test case set, collecting input data and state data after execution of each test case; and evaluating each index, including data processing, determining an input domain, evaluating index calculation and visual display and the like. The method is specially used for evaluating agent test cases based on reinforcement learning, can comprehensively reflect the quality of the test cases, overcomes the defects of the existing single-dimension evaluation, meanwhile, the coverage of the test cases is more intuitive, the test personnel can quickly understand and analyze, and the practicability of the evaluation result is improved.

Description

Technical Field

[0001] This invention relates to the fields of reinforcement learning and agent testing, and in particular to a method for evaluating the multi-dimensional coverage of test cases for reinforcement learning agents based on PCA dimensionality reduction. Background Technology

[0002] Reinforcement learning-based agents have been widely applied across various fields. To ensure the performance and reliability of these agents, effective testing is crucial. However, current evaluation methods for reinforcement learning primarily focus on reward values, i.e., the agent's performance, lacking methods for evaluating reinforcement learning test cases. In fact, evaluating test cases should be a key step in the testing process; the quality and coverage of test cases directly determine the effectiveness of reinforcement learning agent testing. Currently, evaluation methods for test cases of reinforcement learning agents have certain limitations: existing technologies often evaluate test cases from only a single dimension, failing to comprehensively reflect the coverage of test cases in the two key spaces of input and state; simultaneously, there is a lack of effective means for processing high-dimensional data, leading to inaccurate evaluation results; furthermore, the presentation of evaluation results is relatively simple and unintuitive, making it difficult for testers to quickly understand the coverage of test cases. Therefore, a new evaluation method for reinforcement learning agent test cases is urgently needed to improve the effectiveness and accuracy of testing. Summary of the Invention

[0003] To overcome the shortcomings of existing methods for evaluating test cases of reinforcement learning agents, such as single evaluation dimensions, poor performance in high-dimensional data processing, and unintuitive evaluation results, this disclosure provides a multi-dimensional coverage evaluation method for reinforcement learning agent test cases based on PCA dimensionality reduction. This method can comprehensively, accurately, and intuitively evaluate the coverage of test cases from multiple dimensions, providing a more scientific and effective basis for testing reinforcement learning agents, thereby promoting the optimization and improvement of reinforcement learning agents.

[0004] This disclosure provides a multi-dimensional coverage evaluation method for reinforcement learning agent test cases based on PCA dimensionality reduction, including: S1, construct evaluation metrics for intelligent agent test cases, including: 2D input space coverage, 3D input space coverage, 2D state space coverage, and 3D state space coverage; S2, Select several test cases for the agent under test as a test case set, and collect the input data and state data after execution for each test case; S3, based on the data obtained in step S2, evaluate each indicator listed in step S1, including: data processing, determining the input domain, calculating the evaluation indicators, and visualization.

[0005] Furthermore, in the evaluation metrics of step S1, the input space refers to the range of all possible input values ​​that can be accepted when the use case is generated; the state space refers to the range of all states experienced when the use case is executed. 2D input space coverage refers to the degree of coverage of the generated test case set in the 2D input space; 3D input space coverage refers to the degree of coverage of the generated test case set in the 3D input space; 2D state space coverage refers to the degree of coverage of the set of states accessed by the generated test case set in the 2D state space during execution. 3D state space coverage refers to the degree of coverage of the set of states accessed by the generated test case set in the 3D state space during execution.

[0006] Furthermore, in step S3, the evaluation method for 2D input space coverage includes: For a given number of test cases, take the input data of the test cases and form a two-dimensional array. Use PCA to reduce each row of data in the two-dimensional array to 2 dimensions. Traverse the two dimensions of the dimensionality-reduced two-dimensional array, find the maximum and minimum values ​​of each dimension, and represent the input domain of PCA; Calculate the coverage of the test case set in the two-dimensional input space, and the coverage ratio of each input feature in the input space; Create a scatter plot to display the input data after PCA processing, mapping the data in each dimension to the range [0,1].

[0007] Furthermore, in step S3, the evaluation method for 3D input space coverage includes: For a given number of test cases, take the input data of the test cases and form a two-dimensional array, then reduce each row of data to three dimensions using the PCA algorithm; Traverse the three dimensions of the dimensionality-reduced 3D array, find the maximum and minimum values ​​of each dimension, and determine the input domain of PCA; Calculate the 3D input space coverage and the coverage of each input feature in the input space; Create a scatter plot to display the input data after PCA processing, with each dimension mapped to [0,1].

[0008] Furthermore, in step S3, the evaluation method for 2D state space coverage includes: Data processing: For a given number of test case execution result data, extract the status data of the test cases to form a two-dimensional array, and then use PCA to reduce each row of data in the two-dimensional array to 2 dimensions; Determine the input domain: Traverse the two dimensions of the dimensionality-reduced two-dimensional array, find the maximum and minimum values, and determine the input domain of PCA; Coverage calculation: Calculate the 2D state space coverage and the coverage of each state feature; Visualization: Create a scatter plot, map the data to [0,1] and display it.

[0009] Furthermore, in step S3, the evaluation method for 3D state space coverage includes: For a given number of test case execution result data, take the state data of the test cases to form a two-dimensional array, and then use PCA to reduce each row of data in the two-dimensional array to 3 dimensions; Iterate through each dimension to find the maximum and minimum values, which represent the input domain of PCA; Calculate the 3D state space coverage and the coverage of each state feature; Create a scatter plot to display the PCA-processed input data, with each dimension mapped to [0,1].

[0010] Furthermore, the method also includes the following steps: After completing the multi-dimensional coverage evaluation of the agent test cases based on the evaluation metrics in step S1, the testers analyze the coverage of the test cases in various scenarios based on the evaluation results, and then optimize and improve the test cases.

[0011] Compared with the prior art, the beneficial effects of this disclosure are: (1) It is specifically designed to evaluate test cases for agents based on reinforcement learning, which can better meet the testing needs of this field and provide more valuable references for the optimization and improvement of agents; (2) By using four indicators—2D input space coverage, 3D input space coverage, 2D state space coverage, and 3D state space coverage—the coverage of test cases can be comprehensively evaluated from two spaces (input and state) and two dimensions (2D and 3D). This can more comprehensively reflect the quality of test cases and overcome the shortcomings of existing single-dimensional evaluation. (3) By creating a scatter plot to display the input data after PCA processing, and mapping each dimension to [0,1], the coverage of test cases becomes more intuitive, making it easier for testers to quickly understand and analyze, and improving the practicality of the evaluation results; (4) It is easy to organize and implement, and helps to improve the performance and reliability of reinforcement learning agents. Attached Figure Description

[0012] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments of this disclosure taken in conjunction with the accompanying drawings, in which the same reference numerals generally represent the same components.

[0013] Figure 1 A flowchart illustrating an exemplary embodiment of this disclosure; Figure 2 Example of generating a map for 2D input spatial coverage; Figure 3 Example of generating a 3D state space coverage graph. Detailed Implementation

[0014] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0015] This disclosure provides a method for evaluating the multi-dimensional coverage of test cases for reinforcement learning agents based on PCA dimensionality reduction. The process is shown in the attached figure. Figure 1 As shown, this method mainly includes the following four evaluation indicators and corresponding testing methods: (1) 2D input space coverage Definition: Input space refers to the range of all possible input values ​​that can be accepted when generating test cases. 2D input space coverage refers to the degree of coverage of the generated test case set in the 2D input space. Testing Method: For a given number of test cases, the input data of the test cases are taken to form a two-dimensional array. Then, each row of data in the two-dimensional array is reduced to 2 dimensions using PCA. The maximum and minimum values ​​of each dimension are found to represent the input domain of PCA. Then, the coverage of the input space and the coverage of each feature are calculated, and a scatter plot is created to display the input data after PCA processing, with each dimension mapped to [0,1].

[0016] (2) 3D input space coverage Definition: Input space refers to the range of all possible input values ​​that can be accepted when generating test cases. 3D input space coverage refers to the degree of coverage of the generated test case set in the 3D input space. Testing Method: For a given number of test cases, the input data of the test cases are formed into a two-dimensional array. Then, each row of data in the two-dimensional array is reduced to three dimensions using PCA. The maximum and minimum values ​​of each dimension are found to represent the input domain of PCA. Then, the coverage of the input space and the coverage of each feature are calculated, and a scatter plot is created to display the input data after PCA processing, with each dimension mapped to [0,1].

[0017] (3) 2D state space coverage Definition: State space refers to the range of all states experienced during test case execution. 2D state space coverage refers to the degree of coverage of the set of states accessed by the generated test case set during execution in the 2D state space. Test Method: For a given number of test case execution results, the state data of the test cases are taken to form a two-dimensional array. Then, each row of data in this two-dimensional array is reduced to 2 dimensions using PCA. The maximum and minimum values ​​of each dimension are found to represent the input domain of PCA. Then, the coverage of the state space and the coverage of each feature are calculated, and a scatter plot is created to display the input data after PCA processing. Each dimension is mapped to [0,1].

[0018] (4) 3D state space coverage Definition: State space refers to the range of all states experienced during test case execution. 3D state space coverage refers to the degree of coverage of the set of states accessed by the generated test case set during execution in the 3D state space. Testing Method: For a given number of test case execution results, the state data of the test cases are taken to form a two-dimensional array. Then, each row of data in this two-dimensional array is reduced to 3 dimensions using PCA. The maximum and minimum values ​​of each dimension are found to represent the input domain of PCA. Then, the coverage of the state space and the coverage of each feature are calculated, and a scatter plot is created to display the input data after PCA processing. Each dimension is mapped to [0,1].

[0019] This embodiment takes a Pac-Man-based agent based on reinforcement learning as an example, and evaluates its test cases using the evaluation method of this invention. The specific steps are as follows: (1) Data collection 1000 test cases were selected for this Pac-Man agent, and the input data and post-execution state data for each test case were collected. The input data may include the initial coordinates of Pac-Man's position, the initial coordinates of the ghost's position, and the distribution density of the dots; the state data may include Pac-Man's position at different times, score, remaining lives, and the real-time position of the ghost.

[0020] (2) Calculation of 2D input space coverage 1) Data processing: The input data of 1000 test cases are organized into a 1000×n two-dimensional array (n is the number of input features, such as Pac-Man's initial x-coordinate, initial y-coordinate, Ghost 1's initial x-coordinate, etc.). The PCA algorithm is used to reduce each row of data to 2 dimensions, resulting in a 1000×2 two-dimensional array. 2) Determine the input domain: Traverse the two dimensions of the dimensionality-reduced two-dimensional array and find the maximum and minimum values ​​of each dimension to determine the input domain of PCA. For example, if the maximum value of the first dimension is 10 and the minimum value is 2, and the maximum value of the second dimension is 8 and the minimum value is 1, then the input domain is the range of [2,10] for the first dimension and [1,8] for the second dimension. 3) Coverage calculation: Calculate the coverage of the generated test case set in the 2D input space, i.e., the 2D input space coverage. At the same time, calculate the coverage of each input feature, such as the coverage ratio of the feature of Pac-Man's initial position coordinates in the input space. 4) Visualization: Create a scatter plot to display the input data after PCA processing, mapping the data of each dimension to the range [0,1]. For example, map the first dimension [2,10] to [0,1], where 2 corresponds to 0, 10 corresponds to 1, and intermediate values ​​are converted proportionally. The second dimension is similar.

[0021] (3) Calculation of 3D input space coverage 1) Data processing: Using the same two-dimensional array composed of the input data of the above 1000 test cases, the PCA algorithm is used to reduce each row of data to 3 dimensions, resulting in a 1000×3 two-dimensional array. 2) Determine the input domain: Traverse the three dimensions of the dimensionality-reduced three-dimensional array, find the maximum and minimum values ​​of each dimension, and determine the input domain of PCA. 3) Coverage calculation: Calculate the coverage of the 3D input space and the coverage of each input feature, such as the coverage of the ghost's initial position coordinates feature. 4) Visualization: Create a scatter plot to display the processed data, with each dimension mapped to [0,1].

[0022] (4) Calculation of 2D state space coverage 1) Data processing: Take the state data of the execution results of 1000 test cases and form a 1000×m two-dimensional array (m is the number of state features, such as Pac-Man's x-coordinate at a certain time, y-coordinate at a certain time, current score, etc.). Use the PCA algorithm to reduce each row of data to 2 dimensions to obtain a 1000×2 two-dimensional array. 2) Determine the input domain: Traverse the two dimensions of the dimensionality-reduced two-dimensional array, find the maximum and minimum values, and determine the input domain of PCA. 3) Coverage calculation: Calculate the 2D state space coverage and the coverage of each state feature, such as the coverage ratio of the state feature of Pac-Man's position. 4) Visualization: Create a scatter plot, map the data to [0,1] and display it.

[0023] (5) Calculation of 3D state space coverage 1) Data processing: The above state data is reduced to 3 dimensions using PCA to obtain a 1000×3 two-dimensional array. 2) Determine the input domain: Traverse the three dimensions and find the maximum and minimum values ​​to determine the input domain.

[0024] Coverage calculation: Calculate the 3D state space coverage and the coverage of each state feature, such as the coverage of the state feature of score. 3) Visualization: Create a scatter plot, with each dimension mapped to [0,1] for display. Through the above steps, a multi-dimensional coverage evaluation of the Pac-Man agent test cases is completed. Testers can analyze the coverage of the test cases in the Pac-Man game scenario based on the evaluation results, and then optimize and improve the test cases, such as adding test cases in certain initial positions or states to improve the performance of the Pac-Man agent.

[0025] The above technical solutions are merely exemplary embodiments of the present invention. For those skilled in the art, based on the application methods and principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the methods described in the specific embodiments of the present invention. Therefore, the methods described above are merely preferred and not restrictive.

Claims

1. A method for evaluating the multi-dimensional coverage of intelligent agent test cases based on PCA dimensionality reduction, characterized in that, Includes the following steps: S1, construct evaluation metrics for intelligent agent test cases, including: 2D input space coverage, 3D input space coverage, 2D state space coverage, and 3D state space coverage; S2, Select several test cases for the agent under test as a test case set, and collect the input data and state data after execution for each test case; S3, based on the data obtained in step S2, evaluate each indicator listed in step S1, including: data processing, determining the input domain, calculating the evaluation indicators, and visualization.

2. The method according to claim 1, characterized in that, In the evaluation metrics of step S1, the input space refers to the range of all possible input values ​​that can be accepted when the test case is generated; the state space refers to the range of all states that the test case experiences during execution. 2D input space coverage refers to the degree of coverage of the generated test case set in the 2D input space; 3D input space coverage refers to the degree of coverage of the generated test case set in the 3D input space; 2D state space coverage refers to the degree of coverage of the set of states accessed by the generated test case set in the 2D state space during execution. 3D state space coverage refers to the degree of coverage of the set of states accessed by the generated test case set in the 3D state space during execution.

3. The method according to claim 2, characterized in that, In step S3, the evaluation method for 2D input space coverage includes: For a given number of test cases, take the input data of the test cases and form a two-dimensional array. Use PCA to reduce each row of data in the two-dimensional array to 2 dimensions. Traverse the two dimensions of the dimensionality-reduced two-dimensional array, find the maximum and minimum values ​​of each dimension, and represent the input domain of PCA; Calculate the coverage of the test case set in the two-dimensional input space, and the coverage ratio of each input feature in the input space; Create a scatter plot to display the input data after PCA processing, mapping the data in each dimension to the range [0,1].

4. The method according to claim 2, characterized in that, In step S3, the evaluation method for 3D input space coverage includes: For a given number of test cases, take the input data of the test cases and form a two-dimensional array, then reduce each row of data to three dimensions using the PCA algorithm; Traverse the three dimensions of the dimensionality-reduced 3D array, find the maximum and minimum values ​​of each dimension, and determine the input domain of PCA; Calculate the 3D input space coverage and the coverage of each input feature in the input space; Create a scatter plot to display the input data after PCA processing, with each dimension mapped to [0,1].

5. The method according to claim 2, characterized in that, In step S3, the evaluation method for 2D state space coverage includes: Data processing: For a given number of test case execution result data, extract the status data of the test cases to form a two-dimensional array, and then use PCA to reduce each row of data in the two-dimensional array to 2 dimensions; Determine the input domain: Traverse the two dimensions of the dimensionality-reduced two-dimensional array, find the maximum and minimum values, and determine the input domain of PCA; Coverage calculation: Calculate the 2D state space coverage and the coverage of each state feature; Visualization: Create a scatter plot, map the data to [0,1] and display it.

6. The method according to claim 2, characterized in that, In step S3, the evaluation method for 3D state space coverage includes: For a given number of test case execution result data, take the state data of the test cases to form a two-dimensional array, and then use PCA to reduce each row of data in the two-dimensional array to 3 dimensions; Iterate through each dimension to find the maximum and minimum values, which represent the input domain of PCA; Calculate the 3D state space coverage and the coverage of each state feature; Create a scatter plot to display the PCA-processed input data, with each dimension mapped to [0,1].

7. The method according to any one of claims 1-5, characterized in that, It also includes the following steps: After completing the multi-dimensional coverage evaluation of the agent test cases based on the evaluation metrics in step S1, the testers analyze the coverage of the test cases in various scenarios based on the evaluation results, and then optimize and improve the test cases.

Images