Limit test method, system and equipment of artificial intelligence model and medium

Abstract

The invention provides a limit test method, system and device for an artificial intelligence model and a medium, and the method specifically comprises the steps: constructing a plurality of scene templates for different types of games, and generating a test case set by combining and transforming the scene templates according to a boundary condition input by a user; based on the test case set, according to a preset optimization objective function, adopting a heuristic search algorithm to determine an optimal test parameter combination meeting the artificial intelligence model; performing automatic testing on the artificial intelligence model according to the optimal test parameter combination, and generating a model test result; and analyzing the model test result, determining the performance of the artificial intelligence model under the boundary condition, generating a model test report, and carrying out optimization and re-test on the artificial intelligence model through the model test report. According to the method, full-process automation and intellectualization of game AI model testing are achieved, and the testing efficiency and the model quality are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a limit testing method, system, device and medium for an artificial intelligence model. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, especially the widespread application of large language models (such as Claude, GPT-4) and image processing models (such as Haiper), the game development industry is undergoing unprecedented changes. These advanced AI models are integrated into multiple self-developed systems such as game customer service systems, translation platforms, and art image generation, greatly improving the efficiency and quality of game development, operations, and user experience. They can not only handle complex natural language interactions and generate high-quality image content, but also achieve seamless multi-language translation, providing strong support for the global game market.

[0003] However, although these AI models have demonstrated excellent performance under standard testing environments, their stability and reliability face severe challenges when faced with extreme or boundary conditions. The gaming environment is complex and changeable, and player behavior is difficult to predict. These extreme conditions may include high concurrent requests, abnormal input, resource constraints, and other scenarios, all of which place higher demands on the robustness of AI models.

[0004] Currently, boundary testing for AI models mainly relies on manually designed test cases, which is not only time-consuming and laborious, but also difficult to fully cover all possible extreme situations, resulting in low test efficiency and easy omission of important issues. In addition, with the iterative updates of game versions, new boundary conditions continue to emerge, and traditional manual testing methods are difficult to keep up with this speed of change, further exacerbating the difficulty and complexity of testing work. Summary of the invention

[0005] The purpose of the present invention is to provide an extreme testing method, system, equipment and medium for an artificial intelligence model, which realizes the full process automation and intelligence of game AI model testing, significantly improves the testing efficiency and model quality, and provides a strong guarantee for the stability of game products and user experience, so as to solve at least one of the above-mentioned prior art problems.

[0006] In a first aspect, the present invention provides a limit testing method for an artificial intelligence model, the method specifically comprising:

[0007] Build multiple scenario templates for different types of games, and generate a set of test cases by combining and transforming each scenario template according to the boundary conditions input by the user;

[0008] Based on the test case set, according to a preset optimization objective function, a heuristic search algorithm is used to determine the optimal test parameter combination that satisfies the artificial intelligence model;

[0009] Performing automated testing on the artificial intelligence model according to the optimal test parameter combination to generate model test results, and monitoring the running status of the artificial intelligence model during the automated testing process by deploying a game data collection component;

[0010] By analyzing the model test results, the performance of the artificial intelligence model under boundary conditions is determined, a model test report is generated, and the artificial intelligence model is tuned and retested based on the model test report.

[0011] In a second aspect, the present invention provides a limit testing system for an artificial intelligence model, the system specifically comprising:

[0012] The first test module is used to construct multiple scene templates for different types of games, and generate a test case set by combining and transforming each scene template according to the boundary conditions input by the user;

[0013] A second testing module is used to determine the optimal test parameter combination that satisfies the artificial intelligence model based on the test case set and according to a preset optimization objective function using a heuristic search algorithm;

[0014] A third testing module is used to perform automated testing on the artificial intelligence model according to the optimal test parameter combination, generate model test results, and monitor the running status of the artificial intelligence model during the automated testing process by deploying a game data collection component;

[0015] The fourth testing module is used to determine the performance of the artificial intelligence model under boundary conditions by analyzing the model testing results, generate a model testing report, and tune and retest the artificial intelligence model according to the model testing report.

[0016] In a third aspect, the present invention provides a computer device comprising: a memory and a processor and a computer program stored in the memory. When the computer program is executed on the processor, it implements the limit testing method of the artificial intelligence model as described in any one of the above methods.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements a limit testing method for an artificial intelligence model as described in any one of the above methods.

[0018] Compared with the prior art, the present invention has at least one of the following technical effects:

[0019] 1. The present invention realizes the full process automation and intelligence of game AI model testing, significantly improves the testing efficiency and model quality, and provides a strong guarantee for the stability of game products and user experience.

[0020] 2. The present invention can automatically generate a variety of test cases, comprehensively cover and test the stability and performance of AI models under extreme conditions, which can not only significantly improve the test efficiency and quality, but also effectively ensure the reliability and security of the game system, providing players with a more stable and smoother gaming experience.

[0021] 3. The present invention provides an efficient and comprehensive AI model limit testing method for the game development industry, which helps to improve the reliability and security of the game system and promote the in-depth application and development of AI technology in the game field.

[0022] 4. The present invention automatically generates test cases, reduces manual intervention, and significantly shortens the test cycle.

[0023] 5. The present invention generates diversified test cases by combining and transforming scenario templates, comprehensively covering possible extreme situations.

[0024] 6. The present invention determines the optimal test parameter combination based on a heuristic search algorithm to ensure maximum test effect.

[0025] 7. The present invention deploys game data collection components to monitor the running status of the AI ​​model in real time, and performs optimization and retesting according to the test results to improve model performance.

[0026] 8. The present invention combines player portrait data and deep reinforcement learning algorithms to perform personalized optimization and adjustment on the AI ​​model to enhance user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 It is a flowchart of a limit testing method of an artificial intelligence model provided by one embodiment of the present invention;

[0029] Figure 2 It is a structural schematic diagram of a limit testing system for an artificial intelligence model provided by one embodiment of the present invention;

[0030] Figure 3 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0032] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0033] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0034] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0035] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0036] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0037] In the embodiment of the present application, the execution subject of the process includes a terminal device, which includes but is not limited to: a server, a computer, a smart phone, a tablet computer, and other devices capable of executing the method disclosed in the present application. Figure 1 A schematic flow chart of a limit testing method for an artificial intelligence model disclosed in an embodiment of the present invention is shown, and is described in detail as follows:

[0038] S101, construct multiple scene templates for different types of games, and generate a test case set by combining and transforming each scene template according to the boundary conditions input by the user.

[0039] In this embodiment, according to the boundary conditions input by the user, the system can automatically select and combine the corresponding scene templates, and transform the elements in the templates to meet specific needs. For example: in an RPG game, according to the condition that "the player level is 10 and the task type is to find treasure", the system selects the "dungeon" scene template, adds monsters and treasure chests that match the player level, and adjusts the maze difficulty to meet the task requirements. In a shooting game, if the user requires "night battle on city streets", the system selects the "city streets" scene template, adjusts the lighting effect to night mode, and adds enemy types and tactical strategies that are unique to the night.

[0040] By combining and transforming scenario templates, the system can generate a series of specific test case sets. Each test case contains detailed information such as scenario description, player status, task objectives, expected results, etc. These test cases can be used for subsequent automated testing or manual testing to ensure that the game can run normally and meet user needs in different scenarios and conditions.

[0041] In this embodiment, by constructing a variety of scenario templates and combining and transforming them according to the boundary conditions input by the user, a large number of test cases can be generated, thereby significantly improving the coverage and comprehensiveness of the test.

[0042] In some embodiments, in the above step S101, the step of constructing multiple scene templates for different types of games and generating a test case set by combining and transforming each scene template according to the boundary conditions input by the user specifically includes:

[0043] Constructing multiple scene templates for different types of games to form a scene template library, wherein the scene templates include several key elements and several constraints;

[0044] Acquire boundary condition data input by a user, match the boundary condition data with each scene template in the scene template library, and extract a plurality of first scene templates;

[0045] Using a depth-first exploration algorithm, traverse the combination schemes between the key elements and constraints of each first scenario template to generate an initial test case set;

[0046] A greedy algorithm is used to analyze the coverage and differences of each initial test case in the initial test case set to obtain a target test case set.

[0047] In this embodiment, several different types of games are first defined, such as role-playing games (RPGs), shooting games, strategy games, etc. Each game type has its own unique elements and gameplay, for example: RPG scene template, key elements include character level, equipment type, monster type, map type, etc., and constraints include that the character level must be higher than the monster level to cause damage, the equipment type affects the character attributes, and specific maps have specific monsters and events, etc.; shooting game scene template, key elements include weapon type, enemy type, terrain (such as open areas, buildings), etc., and constraints include that different weapons have different damage effects on enemies, and certain terrains will affect shooting accuracy and concealment, etc.; strategy game scene template, key elements include resource types (such as gold coins, wood), unit types, terrain (such as plains, mountains), etc., and constraints include that resource allocation affects unit production and upgrades, and different terrains affect unit movement speed and combat effectiveness, etc.

[0048] The above scene templates are stored in a database to form a scene template library, and each template has a unique identifier and description information.

[0049] The user inputs specific boundary conditions of the game through the interface, such as "In the RPG game, the character level is 10, equipped with a primary sword, and facing an intermediate monster in the forest." The boundary conditions entered by the user are matched with each template in the scene template library, and the scene template that meets the conditions is extracted as the first scene template. Starting from the first scene template, the depth-first search (DFS) algorithm is used to traverse all possible combinations of key elements and constraints. For example, considering different combinations of character levels, equipment, and monsters, multiple initial test cases are generated.

[0050] A greedy algorithm is applied to the initial test case set, with the goal of maximizing the coverage and diversity of the test cases. Coverage analysis is used to ensure that the test cases can cover as much game logic and scenarios as possible, and diversity analysis is used to reduce redundancy between test cases and ensure that each test case provides unique information. Finally, a streamlined and efficient target test case set is obtained.

[0051] In this embodiment, by building a scene template library and an automatic matching mechanism, test cases for specific game types and boundary conditions are quickly generated, reducing the time and cost of manually writing test cases. The depth-first exploration algorithm ensures that the test cases can cover complex game logic and scenarios, while the optimization of the greedy algorithm further improves the comprehensiveness and depth of the test. By maximizing the coverage and diversity of test cases, it is ensured that the test can discover more potential problems and defects, and improve the quality and stability of the game.

[0052] S102, based on the test case set and in accordance with a preset optimization objective function, a heuristic search algorithm is used to determine an optimal test parameter combination that satisfies the artificial intelligence model.

[0053] In this embodiment, first, a set of test cases is constructed and generated according to different scenarios and requirements of the game or software. These test cases contain various possible input conditions and expected outputs, which are used to evaluate the performance and stability of the artificial intelligence model. In order to determine the optimal test parameter combination, an optimization objective function needs to be preset. This function is usually based on the execution results of the test case, such as accuracy, recall rate, F1 score, execution time, etc., to measure the model performance under different parameter combinations.

[0054] Heuristic search algorithms can efficiently search in the solution space to find parameter combinations that are close to the optimal solution. Taking genetic algorithms as an example, their implementation steps may include: 1. Initialize the population: randomly generate a set of parameter combinations as the initial population. 2. Fitness evaluation: use the preset optimization objective function to evaluate the fitness of each parameter combination. 3. Selection operation: select excellent parameter combinations for genetic operations based on fitness. 4. Crossover and mutation: perform crossover and mutation operations on the selected parameter combinations to generate new parameter combinations. 5. Iterative optimization: repeat steps 2-4 until the stopping condition is met (such as reaching the maximum number of iterations, fitness is no longer significantly improved, etc.). After multiple iterations, the algorithm will output a set of parameter combinations with the highest fitness, that is, the optimal test parameter combination that meets the artificial intelligence model.

[0055] In this embodiment, the heuristic search algorithm can efficiently search in the solution space and quickly find the parameter combination close to the optimal solution, thereby shortening the test cycle. Testing the model under different parameter combinations helps to discover potential errors and unstable factors, thereby enhancing the stability and reliability of the model.

[0056] In some embodiments, in the above step S102, based on the test case set and according to a preset optimization objective function, a heuristic search algorithm is used to determine the optimal test parameter combination that satisfies the artificial intelligence model, specifically including:

[0057] Obtaining game project demand data and specific requirement data of the artificial intelligence model, and constructing an optimization objective function according to the game project demand data and the specific requirement data, wherein the optimization objective function includes model reasoning speed, memory occupancy performance index, and resource consumption index;

[0058] Dividing the test case set into a plurality of parameter subspaces, each parameter subspace representing a class of similar game scenarios;

[0059] In each of the parameter subspaces, a simulated annealing algorithm is used to perform search optimization to obtain multiple test parameter combinations;

[0060] By evaluating the model inference speed, memory usage and resource consumption indicators corresponding to each test parameter combination, it is determined whether the requirements of the optimization objective function are met. If the requirements are met, the test parameter combination is added to the optimal solution set, and the optimal test parameter combination is obtained according to the optimal solution set.

[0061] In this embodiment, it is assumed that an artificial intelligence model is built for a newly developed online multiplayer shooting game for enemy behavior prediction and intelligent path planning. The game project requires the model to have high inference speed, low memory usage, and reasonable resource consumption to ensure the smoothness of the game and the player's experience. The game project requirement data includes that the game needs to support at least 100 players online at the same time, the enemy AI's response time must be less than 50 milliseconds, and the game must have good performance on different devices (including low-end mobile phones, high-end PCs, etc.); the artificial intelligence model specific requirement data includes that the model must be able to handle complex game environments (such as dynamic obstacles, different terrains and weather conditions), the model's accuracy must reach more than 90%, and the model's memory usage must not exceed 10% of the total game memory.

[0062] Based on the above requirements, the optimization objective function is constructed as follows:

[0063]

[0064] Among them, α, β, γ are weight coefficients, which can be adjusted according to project priorities, Inference Speed ​​is the model inference speed, Memory Usage is the memory occupied by the model when running, Total Memory is the total memory of the game, Resource Consumption is the resource consumption when the model is running (such as CPU usage, network bandwidth, etc.), and text Objective is the overall optimization goal of the model.

[0065] The test case set is divided into multiple parameter subspaces according to the game scene characteristics (such as map type, number of players, enemy density, etc.). For example: parameter subspace A: small map, a small number of players, low enemy density; parameter subspace B: large map, a large number of players, high enemy density; parameter subspace C: complex terrain (such as mountains, cities), medium number of players, medium enemy density.

[0066] In each parameter subspace, the simulated annealing algorithm is used to search for the optimal test parameter combination. The simulated annealing algorithm gradually searches for the global optimal solution in the parameter space by simulating the solid annealing process in physics. The specific steps are as follows:

[0067] Initialization: Randomly select a test parameter combination as the current solution and set the initial temperature.

[0068] Iteration: Randomly generate new solutions in the neighborhood of the current solution and calculate the change in the objective function value. If the new solution is better or meets a certain probability (related to the current temperature and the quality of the solution), the new solution is accepted as the current solution.

[0069] Cooling: As the number of iterations increases, the temperature is gradually lowered so that the algorithm gradually converges to the global optimal solution.

[0070] The test parameter combinations obtained by the simulated annealing algorithm in each parameter subspace are evaluated, and the corresponding model inference speed, memory usage, and resource consumption indicators are calculated to determine whether they meet the requirements of the optimization objective function. If they meet the requirements, the test parameter combination is added to the optimal solution set. Finally, the test parameter combination with the best comprehensive performance is selected from the optimal solution set as the final solution.

[0071] In this embodiment, by optimizing the construction of the objective function and the application of the simulated annealing algorithm, a test parameter combination that performs well in terms of inference speed, memory usage, and resource consumption can be found, thereby improving the overall performance of the model.

[0072] Further, obtaining the optimal test parameter combination according to the optimal solution set specifically includes:

[0073] Using K-means clustering algorithm, the optimal solution set is divided into K clusters;

[0074] Selecting a cluster center point from each of the clusters as a representative sample point to form a sample subset;

[0075] For each of the sample points, a boundary value analysis method is used to generate corresponding boundary value test data according to the game state parameters and event triggering conditions;

[0076] For numeric type parameters, select the upper and lower limits as boundary values; for Boolean type parameters, select true and false as boundary values; for enumeration type parameters, select all enumeration values ​​as boundary values;

[0077] Using an orthogonal table design method, constructing an orthogonal experimental group from the sample subset and the boundary value test data, each factor of the orthogonal experimental group represents a game state parameter or an event triggering condition;

[0078] A test parameter combination satisfying orthogonality is selected from the orthogonal test group, and the test parameter combination satisfying orthogonality is used as the optimal test parameter combination.

[0079] In this embodiment, through further clustering, sample selection, boundary value analysis and orthogonal table design, a set of test parameter combinations that are both representative and efficient are generated to comprehensively verify the stability and performance of the game under different conditions.

[0080] Each test parameter combination in the optimal solution set is regarded as a data point and clustered using the K-means algorithm. The algorithm iteratively assigns data points to the nearest cluster center and updates the location of the cluster center until the stopping condition is met (such as the cluster center no longer changes or the maximum number of iterations is reached). In each cluster, the cluster center point is selected as the representative sample point of the cluster. These center points are usually located in the center of the cluster and can better reflect the common characteristics of the test parameter combinations in the cluster.

[0081] Boundary value analysis method is used to generate boundary value test data. For numerical type parameters, for example, if a parameter in the game is "player speed", its normal range is [1,10], then 1 (lower limit) and 10 (upper limit) are selected as boundary value test data. For Boolean type parameters, such as "whether to enable automatic aiming", true and false are directly selected as boundary values. For enumeration type parameters, such as "weapon type" contains {pistol, rifle, sniper rifle}, then all enumeration values ​​are selected as boundary value test data.

[0082] Each cluster center (representing a different game scene or configuration) and the boundary value test data obtained through boundary value analysis are considered as factors of the orthogonal experiment. The level of each factor is the different values ​​of the factor (i.e., the specific parameter combination or boundary value of the cluster center). According to the number of factors and the number of levels of each factor, select a suitable orthogonal table. For example, if there are 3 factors and each factor has 2 levels (such as two cluster centers and their boundary value tests), you can choose L4 (23 to represent 3 factors and 2 levels for each factor. According to the arrangement of the orthogonal table, select the test parameter combination from the sample subset and the boundary value test data to construct the orthogonal experiment group. Each row of the orthogonal table represents a test parameter combination, which ensures that the different levels of each factor appear evenly in the experiment, while reducing the number of experiments.

[0083] Since the design of the orthogonal table has ensured the orthogonality of the experiment (i.e., the independence of factors), all test parameter combinations can be directly selected from the orthogonal test group as the optimal test parameter combinations. These combinations not only cover different game scenarios and configurations, but also take into account boundary conditions, so they have high representativeness and test efficiency.

[0084] In this embodiment, cluster analysis and orthogonal array design can ensure that the test parameter combination covers various possible scenarios and boundary conditions of the game, thereby improving the comprehensiveness and coverage of the test. Orthogonal array design reduces the test cost by reducing the number of experiments while maintaining the effectiveness and representativeness of the test.

[0085] S103, performing automated testing on the artificial intelligence model according to the optimal test parameter combination to generate model test results, and at the same time monitoring the operating status of the artificial intelligence model during the automated testing process by deploying a game data collection component.

[0086] In this embodiment, the test environment is configured according to the optimal test parameter combination, including hardware resources, software version, network conditions, etc., and automated test frameworks and tools such as Selenium, Appium, etc. are deployed to execute test cases. The test case set generated based on the optimal test parameter combination is loaded into the automated test framework to ensure that the test cases cover the main functions and boundary conditions of the model, start the automated test framework, and execute the test cases according to the preset test sequence. Monitor the test execution process and record test logs and exception information. After the test is completed, collect test data, including test time, execution results (success / failure), performance indicators (such as response time, accuracy), etc. Perform statistical analysis on the test results and generate a test report.

[0087] Select appropriate game data collection components according to test requirements, such as APM (Application Performance Management) tools, log collection systems, etc. Deploy data collection components in the test environment, and configure their monitoring indicators and collection frequency to ensure that the data collection components can collect key data in real time and accurately during the model operation process. Start the data collection component and start monitoring the running status of the model during the automated testing process. View monitoring data in real time, including key indicators such as CPU usage, memory usage, response time, error rate, etc., and automatically trigger alarms when monitoring data exceeds the preset threshold.

[0088] In this embodiment, automated testing can significantly reduce the workload of manual testing and improve testing efficiency. At the same time, test cases based on the optimal test parameter combination can more accurately evaluate model performance. Through automated testing and detailed test result analysis, potential problems and defects in the model can be discovered to ensure test quality.

[0089] In some embodiments, in the above step S103, the automatic testing of the artificial intelligence model according to the optimal test parameter combination to generate the model test result specifically includes:

[0090] Use JMeter to simulate a high-concurrency request test environment, perform automated testing on the artificial intelligence model using the optimal test parameter combination in the high-concurrency request test environment, and obtain a first model test result;

[0091] Selenium is used to simulate a user interaction test environment, and the artificial intelligence model is automatically tested using the optimal test parameter combination in the user interaction test environment to obtain a second model test result.

[0092] In this embodiment, JMeter and Selenium are used to simulate a high concurrent request test environment and a user interaction test environment respectively, and automated testing is performed using the optimal test parameter combination determined previously.

[0093] Specifically, create a new test plan in JMeter, add a thread group, set the number of threads (i.e., the number of concurrent users) and the number of loops to simulate a high concurrent request scenario. For example, setting the number of threads to 1000 and the number of loops to 5 means simulating 1000 users to send 5 requests at the same time. Add HTTP request defaults in the thread group, and set basic information such as the server name or IP address, port number, etc. Add an HTTP request, enter the API interface URL of the chatbot, and set the request method (such as POST) and request body (including the optimal test parameter combination, such as user questions, conversation context, etc.). Add listeners such as "Aggregate Report" or "Graphical Results" to view and analyze the test results after the test is completed. Then start the test plan, and JMeter will simulate a large number of users sending requests to the chatbot at the same time. After the test is completed, view performance indicators such as response time, throughput, and error rate through the listener as the first model test result.

[0094] Install Selenium WebDriver (such as ChromeDriver) and Selenium libraries (such as Selenium Python bindings), and prepare a test script writing environment (such as Python IDE). Use Selenium to write automated test scripts to simulate user login, open a chat window, enter questions (use user questions in the optimal test parameter combination), receive and verify replies, and other interactive processes. You can use Selenium's WebDriver API to simulate user clicks, enter text, wait for elements to load, and other operations. Then execute the test script, and Selenium will automatically open the browser to simulate the user's interaction with the chatbot. During the test, you can capture and verify the chatbot's reply content, response time, etc. as the second model test result. You can use assertions to check whether the reply is as expected and record any failures.

[0095] In this embodiment, the high-concurrency test of JMeter can be used to evaluate the performance of the chatbot under high load, including key indicators such as response time, throughput, and error rate. The user interaction test of Selenium can be used to simulate real user scenarios and evaluate the chatbot's performance in terms of user experience, such as conversation fluency and accuracy.

[0096] S104, determining the performance of the artificial intelligence model under boundary conditions by analyzing the model test results, generating a model test report, and re-tuning the artificial intelligence model based on the model test report.

[0097] In this embodiment, the data collected from the automated testing process includes model output and performance indicators under boundary conditions (such as extreme inputs, resource constraints, etc.). Identify which boundary conditions have a significant impact on model performance, analyze key indicators such as the model's output error rate, response time, and resource consumption under these boundary conditions, and use statistical methods and machine learning algorithms (such as cluster analysis and regression analysis) to deeply understand the relationship between boundary conditions and model performance. Based on the analysis results, determine under which boundary conditions the model performs poorly, i.e., the problem areas. Analyze the causes of the problem areas, such as uneven data distribution, insufficient feature extraction, and model structure defects.

[0098] Organize the test data, analysis results and problem areas into a structured report. The report includes test objectives, test methods, test environment, test results (including performance under boundary conditions), problem area analysis, improvement suggestions, etc. You can also use visualization tools such as charts and tables to clearly display the test results and analysis conclusions.

[0099] According to the improvement suggestions in the test report, formulate a specific model tuning plan, which may include data cleaning and preprocessing, model structure optimization, parameter adjustment, introduction of new features, etc. Modify and adjust the model according to the tuning plan, rerun the test cases in the same test environment, and focus on the boundary conditions of the problem area. Analyze the test results after tuning to verify whether the performance of the model under boundary conditions has been improved. If the tuning effect is not ideal, continue to adjust the tuning plan and repeat the tuning and retesting process.

[0100] In this embodiment, by deeply analyzing the test results and the performance under boundary conditions, it is possible to discover the problems in the model and make targeted adjustments, thereby improving the overall performance of the model. Boundary condition testing helps to identify the performance of the model under extreme conditions, and through tuning, the stability and robustness of the model can be enhanced.

[0101] In some embodiments, in the above step S104, the tuning and retesting of the artificial intelligence model through the model test report specifically includes:

[0102] Determining specific layers of the artificial intelligence model that require tuning based on the pattern test report;

[0103] Determine an original training data set according to the specific layer, and expand and improve the original training data set through a generative adversarial network to form a target training data set;

[0104] Based on the target training data set, transfer learning and model fine-tuning are performed on the artificial intelligence model.

[0105] In this example, the test reports generated by JMeter and Selenium are carefully analyzed, especially those test cases with long response time, high error rate or poor user feedback. By comparing the input and output of these test cases and combining the model architecture knowledge, we can determine which specific layer or layers the performance bottleneck is located in, and accurately locate the specific layer that needs to be tuned according to the model architecture documentation and source code.

[0106] Collect and organize the original data sets used to train artificial intelligence models. For example, taking the game guide robot as an example, these data sets should contain multiple types of questions and corresponding responses. Then design a GAN model suitable for the game guide robot data set, including a generator and a discriminator. The generator is responsible for generating fake question-response pairs, which should be as close to the distribution of the real data set as possible, and the discriminator is responsible for distinguishing the generated data from the real data. Train the GAN model so that the generator can generate question-response pairs that are closer and closer to the real data. Use the new data generated by the generator to expand the original training data set to form the target training data set. These data can increase the generalization ability and robustness of the model.

[0107] Set up a new training environment, including the necessary hardware resources (such as a GPU), software frameworks (such as TensorFlow or PyTorch), and dependent libraries. Load the previously trained game-guided robot, including its weights and architecture. Perform transfer learning on the model using the target training dataset, focusing on adjusting specific layers that have been identified as requiring tuning. These layers can be fine-tuned using a smaller learning rate and more iterations to avoid damaging the performance of the model in other areas. At the same time, the weights of other parts of the model can be kept unchanged or fine-tuned to keep the overall performance stable. Use a new test set (which can be different from the original test set) to verify and evaluate the fine-tuned model, comparing the model performance before and after fine-tuning to ensure that the tuning effect is as expected.

[0108] In this embodiment, using a generative adversarial network to expand the training data set can increase the diversity of the model's training samples, thereby improving the model's generalization ability and robustness. By fine-tuning rather than retraining the entire model, the risk of overfitting due to insufficient data or uneven data distribution can be reduced. Transfer learning and fine-tuning techniques can accelerate the model tuning process and reduce the time and resources required to train a model from scratch.

[0109] In some embodiments, the method further comprises:

[0110] Obtain the player's operation logs in different game scenarios, analyze the operation logs through the LSTM model, and obtain player portrait data, where the player portrait data includes player behavior patterns and player preference characteristics;

[0111] The game level pass rate, key indicators of player retention time and player experience data are used as reward functions, the player portrait data is used as the state space, and the interactive behavior of the artificial intelligence model is used as the action space. The artificial intelligence model is personalized and optimized through a deep reinforcement learning algorithm.

[0112] In this embodiment, the operation logs of players in different game scenarios are collected through the game client or server. These logs include detailed operation records such as player clicks, movements, skill releases, item usage, and corresponding timestamps and game scene identifiers. The collected operation logs are converted into structured data, including fields such as player ID, timestamp, operation type, operation object, scene ID, etc. Key features are extracted from the operation logs, such as operation frequency, operation sequence, skill usage preference, item usage habits, etc. The player's operation logs are arranged in chronological order to form time series data as the input of the LSTM model.

[0113] Use Python's Keras or TensorFlow library to build an LSTM model. The model includes an input layer, multiple LSTM layers (for capturing long-term dependencies in time series), a fully connected layer (for feature extraction and classification / regression), and an output layer. Set appropriate hyperparameters, such as the number of units in the LSTM layer, learning rate, optimizer, etc. Use preprocessed operation log data to train the LSTM model. Evaluate the performance of the model through methods such as cross-validation to ensure that the model can accurately capture the player's behavior patterns and preference characteristics. Use the trained LSTM model to analyze new operation logs and generate player portrait data, which includes the player's behavior patterns (such as operation habits, skill use strategies, etc.) and preference characteristics (such as favorite game types, role positioning, etc.).

[0114] Design a reasonable reward mechanism to encourage AI models to take actions that improve player experience and game performance. Use game level pass rate, player retention time key indicators, and player experience data (such as satisfaction ratings, feedback, etc.) as components of the reward function. Use player portrait data as the state space, including player behavior patterns and preference characteristics. Define the interactive behavior of the AI ​​model as the action space, including in-game decision-making, recommendations, guidance, etc. Use deep reinforcement learning algorithms (such as DQN, A3C, etc.) to train the AI ​​model. Through interaction with the simulated environment or the actual game environment, continuously optimize the model's strategy and behavior to maximize the expected value of the reward function.

[0115] In this embodiment, the player's operation log is analyzed by the LSTM model to generate personalized player portrait data to provide players with a gaming experience that is more in line with their interests and preferences. The artificial intelligence model can adjust the game difficulty, recommended content, etc. according to the player's real-time behavior, thereby improving the player's satisfaction and retention rate. The deep reinforcement learning algorithm can continuously optimize the strategy and behavior of the artificial intelligence model, and improve key indicators such as the game level pass rate and player experience data. Through continuous training and optimization, the artificial intelligence model can gradually adapt to the needs and changes of different players and maintain the competitiveness and attractiveness of the game. The artificial intelligence model can intelligently interact and guide according to the player's behavior patterns and preference characteristics, enhancing the interactivity and fun of the game. Through real-time interaction and feedback loops with players, the artificial intelligence model can continue to learn and evolve, bringing players a richer gaming experience.

[0116] Reference Figure 2 An embodiment of the present invention provides a limit test system 2 for an artificial intelligence model, and the system 2 specifically includes:

[0117] The first test module 201 is used to construct multiple scene templates for different types of games, and generate a test case set by combining and transforming each scene template according to the boundary conditions input by the user;

[0118] The second testing module 202 is used to determine the optimal test parameter combination that satisfies the artificial intelligence model based on the test case set and according to a preset optimization objective function by using a heuristic search algorithm;

[0119] The third testing module 203 is used to perform an automated test on the artificial intelligence model according to the optimal test parameter combination, generate a model test result, and monitor the running status of the artificial intelligence model during the automated test by deploying a game data collection component;

[0120] The fourth testing module 204 is used to determine the performance of the artificial intelligence model under boundary conditions by analyzing the model testing results, generate a model testing report, and tune and retest the artificial intelligence model according to the model testing report.

[0121] It is understandable that if Figure 1 The contents of the limit test method embodiment of the artificial intelligence model shown in the figure are all applicable to the limit test system embodiment of the artificial intelligence model. The functions specifically implemented by the limit test system embodiment of the artificial intelligence model are the same as those in the example shown in the figure. Figure 1 The limit test method embodiment of the artificial intelligence model shown in FIG. 1 is the same as that of FIG. 1 , and the beneficial effects achieved are the same as those of FIG. Figure 1 The beneficial effects achieved by the embodiment of the limit testing method of the artificial intelligence model shown are also the same.

[0122] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0123] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0124] Reference Figure 3 The embodiment of the present invention further provides a computer device 3, comprising: a memory 302 and a processor 301 and a computer program 303 stored in the memory 302. When the computer program 303 is executed on the processor 301, a limit testing method for an artificial intelligence model as described in any one of the above methods is implemented.

[0125] The computer device 3 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will appreciate that Figure 3 It is only an example of computer device 3 and does not constitute a limitation on computer device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0126] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0127] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may also be an external storage device of the computer device 3, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 3. Further, the memory 302 may also include both an internal storage unit of the computer device 3 and an external storage device. The memory 302 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as the program code of the computer program, etc. The memory 302 may also be used to temporarily store data that has been output or is to be output.

[0128] An embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the limit testing method of the artificial intelligence model as described in any of the above methods is implemented.

[0129] In this embodiment, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0130] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0131] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0132] In the embodiments disclosed in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Claims

1. A limit testing method for an artificial intelligence model, characterized in that: The method specifically comprises: Build multiple scenario templates for different types of games, and generate a set of test cases by combining and transforming each scenario template according to the boundary conditions input by the user; Based on the test case set, according to a preset optimization objective function, a heuristic search algorithm is used to determine the optimal test parameter combination that satisfies the artificial intelligence model; Performing automated testing on the artificial intelligence model according to the optimal test parameter combination to generate model test results, and monitoring the running status of the artificial intelligence model during the automated testing process by deploying a game data collection component; By analyzing the model test results, the performance of the artificial intelligence model under boundary conditions is determined, a model test report is generated, and the artificial intelligence model is tuned and retested based on the model test report.

2. The method according to claim 1, characterized in that The method constructs multiple scene templates for different types of games, and generates a test case set by combining and transforming each scene template according to the boundary conditions input by the user, specifically including: Constructing multiple scene templates for different types of games to form a scene template library, wherein the scene templates include several key elements and several constraints; Acquire boundary condition data input by a user, match the boundary condition data with each scene template in the scene template library, and extract a plurality of first scene templates; Using a depth-first exploration algorithm, traverse the combination schemes between the key elements and constraints of each first scenario template to generate an initial test case set; A greedy algorithm is used to analyze the coverage and differences of each initial test case in the initial test case set to obtain a target test case set.

3. The method according to claim 1, characterized in that Based on the test case set, according to a preset optimization objective function, a heuristic search algorithm is used to determine the optimal test parameter combination that satisfies the artificial intelligence model, specifically including: Obtaining game project demand data and specific requirement data of the artificial intelligence model, and constructing an optimization objective function according to the game project demand data and the specific requirement data, wherein the optimization objective function includes model reasoning speed, memory occupancy performance index, and resource consumption index; Dividing the test case set into a plurality of parameter subspaces, each parameter subspace representing a class of similar game scenarios; In each of the parameter subspaces, a simulated annealing algorithm is used to perform search optimization to obtain multiple test parameter combinations; By evaluating the model inference speed, memory usage and resource consumption indicators corresponding to each test parameter combination, it is determined whether the requirements of the optimization objective function are met. If the requirements are met, the test parameter combination is added to the optimal solution set, and the optimal test parameter combination is obtained according to the optimal solution set.

4. The method according to claim 3, characterized in that: The obtaining of the optimal test parameter combination according to the optimal solution set specifically includes: Using K-means clustering algorithm, the optimal solution set is divided into K clusters; Selecting a cluster center point from each of the clusters as a representative sample point to form a sample subset; For each of the sample points, a boundary value analysis method is used to generate corresponding boundary value test data according to the game state parameters and event triggering conditions; For numeric type parameters, select the upper and lower limits as boundary values; for Boolean type parameters, select true and false as boundary values; for enumeration type parameters, select all enumeration values ​​as boundary values; Using an orthogonal table design method, constructing an orthogonal experimental group from the sample subset and the boundary value test data, each factor of the orthogonal experimental group represents a game state parameter or an event triggering condition; A test parameter combination satisfying orthogonality is selected from the orthogonal test group, and the test parameter combination satisfying orthogonality is used as the optimal test parameter combination.

5. The method according to claim 1, characterized in that The step of automatically testing the artificial intelligence model according to the optimal test parameter combination to generate a model test result specifically includes: Use JMeter to simulate a high-concurrency request test environment, perform automated testing on the artificial intelligence model using the optimal test parameter combination in the high-concurrency request test environment, and obtain a first model test result; Selenium is used to simulate a user interaction test environment, and the artificial intelligence model is automatically tested using the optimal test parameter combination in the user interaction test environment to obtain a second model test result.

6. The method according to claim 1, characterized in that The tuning and retesting of the artificial intelligence model through the model test report specifically includes: Determining specific layers of the artificial intelligence model that require tuning based on the pattern test report; Determine an original training data set according to the specific layer, and expand and improve the original training data set through a generative adversarial network to form a target training data set; Based on the target training data set, transfer learning and model fine-tuning are performed on the artificial intelligence model.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Obtain the player's operation logs in different game scenarios, analyze the operation logs through the LSTM model, and obtain player portrait data, where the player portrait data includes player behavior patterns and player preference characteristics; The game level pass rate, key indicators of player retention time and player experience data are used as reward functions, the player portrait data is used as the state space, and the interactive behavior of the artificial intelligence model is used as the action space. The artificial intelligence model is personalized and optimized through a deep reinforcement learning algorithm.

8. A limit testing system for an artificial intelligence model, characterized in that: The system specifically comprises: The first test module is used to construct multiple scene templates for different types of games, and generate a test case set by combining and transforming each scene template according to the boundary conditions input by the user; A second testing module is used to determine the optimal test parameter combination that satisfies the artificial intelligence model based on the test case set and according to a preset optimization objective function using a heuristic search algorithm; A third testing module is used to perform automated testing on the artificial intelligence model according to the optimal test parameter combination, generate model test results, and monitor the running status of the artificial intelligence model during the automated testing process by deploying a game data collection component; The fourth testing module is used to determine the performance of the artificial intelligence model under boundary conditions by analyzing the model testing results, generate a model testing report, and tune and retest the artificial intelligence model according to the model testing report.

9. A computer device, characterized in that: include: A memory and a processor and a computer program stored in the memory, when the computer program is executed on the processor, implements the limit testing method of the artificial intelligence model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the limit testing method of the artificial intelligence model as described in any one of claims 1 to 7 is implemented.

Images