A reinforcement learning level testing method for game development

Abstract

A reinforcement learning level testing method for the game development stage, characterized by comprising: performing diversity-driven reinforcement learning during the game level testing process, wherein the diversity-driven reinforcement learning algorithm is provided with a task target reward r for the interactive environment. ex and the diversity reward r generated by the diversity evaluation of the current strategy in , where when the diversity of the current strategy is greater than or equal to the threshold δ in And the mission goal reward r ex Less than the threshold δ ex When the learning goal is to focus on obtaining more task target rewards r ex ; When the diversity of the current strategy is less than the threshold δ in When , the learning goal is to focus on obtaining more diversity rewards r in , prompting the agent to choose strategies it had not previously adopted. This invention improves the pertinence of training objectives, enhances the coverage of game environment exploration and the validity of the generated data when using reinforcement learning algorithms for game testing, and better meets the testing needs of industrialized game development pipelines.

Description

Technical Field

[0001] The present invention relates to the application technology of deep reinforcement learning, and in particular to a reinforcement learning level testing method for the game development stage. Background Art

[0002] Game level testing refers to the acceptance and testing of the basic parameters, flow, core gameplay, sensory experience, and other aspects of a game level through the use of testers, automated algorithms, and expert evaluation. It is an important method for preemptively identifying and optimizing issues at all stages of a game project's development and operation, and a crucial means of improving game quality and ensuring user experience. As game systems become increasingly complex, traditional manual testing methods consume increasing amounts of time and money, and for games in the development stage, they also raise confidentiality and security issues. Automated testing methods based on scripts, genetic algorithms, and deep learning are complex to model for changing game environments and require large amounts of data, making their practical application more difficult.

[0003] The application of deep reinforcement learning in the gaming field offers new possibilities for game testing during the R&D phase. Compared to other methods, deep reinforcement learning can continuously interact with the environment to obtain rich information, reducing data pressure. By optimizing the agent's behavior through a manually set reward function, the design pressure shifts from modeling to reward design, significantly lowering the threshold for development and application. The Unity engine provides a plug-in tool, ML-Agent, for accessing reinforcement learning algorithms. Currently, game testing based on deep reinforcement learning has achieved certain results in detecting game texture vulnerabilities, verifying game balance, checking scene collisions, and tracking game performance.

[0004] However, previous deep reinforcement learning-based game level testing techniques often target highly completed, well-developed game levels. During the game development phase, which requires extensive testing, levels are often incomplete and undergo rapid iteration, making them difficult to integrate with current reinforcement learning-based training methods. Furthermore, traditional manual level QA testing typically requires completing a series of sub-goals sequentially and identifying issues along the way. Previous reinforcement learning testing methods, however, focused more on analyzing the data generated by the agent's free exploration throughout the level. This resulted in low overall test coverage for game levels, which typically have a certain logical trigger structure, and a low proportion of valid data to the total data volume.

[0005] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0006] The main purpose of the present invention is to overcome the defects of the above-mentioned background technology and provide a reinforcement learning level testing method for the game development stage.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A reinforcement learning level testing method for the game development stage, characterized by comprising: performing diversity-driven reinforcement learning during the game level testing process, wherein the diversity-driven reinforcement learning algorithm is provided with a task target reward r for the interactive environment. ex and the diversity reward r generated by the diversity evaluation of the current strategy in , where when the diversity of the current strategy is greater than or equal to the threshold δ in And the mission goal reward r ex Less than the threshold δ ex When the learning goal is to focus on obtaining more task target rewards r ex ; When the diversity of the current strategy is less than the threshold δ in When , the learning goal is to focus on obtaining more diversity rewards r in , to encourage the agent to choose a strategy that it has not taken before.

[0009] Further:

[0010] When the diversity of the current strategy is greater than or equal to the threshold δ in And the mission target reward r ex Greater than or equal to the threshold δ ex At any time, the total reward is determined by the task target reward r ex and diversity rewards in The weight of their contribution to the overall reward is determined jointly, and the weight of their contribution to the overall reward is adjusted through hyperparameters.

[0011] The calculation formula of the overall reward function R is:

[0012] R=(1-CMP(r in , δ in )+αCMP(r ex , δ ex ))r in +CMP(r in , δ in )r ex

[0013]

[0014] Among them, CMP(x, y) is a conditional judgment function, (x, y) corresponds to (r in , δ in ) or (r ex , δ ex) or (r in , δ in ), α is a hyperparameter.

[0015] For diversity rewards in , for the current strategy π, randomly sample n behaviors in the entire behavior trajectory and combine them into a behavior vector b n (π), and compare it with the k closest behaviors π′ in the stored strategy library π, and determine the reward based on the calculation of the Euclidean distance, that is:

[0016]

[0017] The method further includes: during model training, maintaining a group of reinforcement learning models to form a strategy library π instead of a single model, and in each round of training, selecting k models π with the largest strategy differences for training.

[0018] The method further includes: forming a unified mount point for centralized management of reward modules, and forming a sparse reward script that can be mounted on any object under unified management, so that when the level designer designs the level process, the sparse reward script can be conveniently mounted on the level interactive object to configure the corresponding task target reward; during training, the unified mount point provides unified management of all mounted reward scripts, thereby simplifying the threshold δ in and δ ex Estimation of and design of reward function.

[0019] The corresponding reward module is dynamically turned on or off according to the agent's exploration progress of the current level to guide the agent to complete a series of sub-goals in the topological order of the level design.

[0020] The method also includes: abstracting and encapsulating the various interactive mechanisms and gameplay elements in the level testing during the game development phase to form a set of standardized test indicators, combining the plug-ins provided by the game engine to set up a quickly configurable reward module and data processing module, and automatically setting the reward function and collecting and processing data by mounting the script on the corresponding Actor and performing preliminary configuration.

[0021] For the reward module, different reward types are defined and corresponding trigger conditions are set to achieve automated management of rewards, and to balance and standardize rewards. The data collection module collects input and location data generated when the reinforcement learning agent runs the intelligent body, and configures monitoring scripts and thresholds to verify the rationality of the data. The data processing module processes and analyzes the collected data to realize location heat maps and provide visualization effects.

[0022] The method also includes: when designing a level process, mounting the corresponding script in a suitable location, and then the test process is automatically carried out. The level designer is not required to re-enter the script until the test process is completed and the model is reviewed and data analyzed.

[0023] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the reinforcement learning level testing method for the game development stage.

[0024] The present invention has the following beneficial effects:

[0025] This paper proposes a reinforcement learning level testing method for the game development stage. It uses a diversity-driven reinforcement learning method to optimize the reinforcement learning training process and integrates a special test reward mechanism into the agent behavior pattern training, which improves the pertinence of the training objectives and is more in line with the goals of game testing.

[0026] The diversity-driven reinforcement learning reward function design method of the present invention improves the coverage of game environment exploration and the validity of the generated data when using reinforcement learning algorithms for game testing, better meeting the testing needs in the industrialized game development pipeline.

[0027] The method of this invention is based on the industrialized pipeline of game development in the industry. Starting from the direction of reward function design and test data analysis, it realizes a set of reinforcement learning test modules that can be quickly integrated into the project and put into use.

[0028] Furthermore, the test indicators used in common level tests during the game development phase were abstracted and encapsulated. Combined with the native ML-Agent plug-in of game development engines such as the Unity engine, a set of quickly configurable reward modules and data processing modules was formed, optimizing the application pipeline.

[0029] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flowchart of a reinforcement learning level testing method for the game development stage according to an embodiment of the present invention.

[0031] Figure 2 This is a mapping relationship diagram of the reward detector according to an embodiment of the present invention.

[0032] Figure 3 This is a workflow diagram of the reward module according to an embodiment of the present invention.

[0033] Figure 4 This is a workflow diagram of the data module according to an embodiment of the present invention.

[0034] Figure 5 This is a diagram of the training algorithm architecture of the reinforcement learning level testing method according to an embodiment of the present invention.

[0035] Figure 6 This is a diagram of the implementation process of the reinforcement learning level testing method according to an embodiment of the present invention.

[0036] Figure 7 A map of the Space Invaders game level environment.

[0037] Figure 8 This is a simple jumping game level map.

[0038] Figure 9 A complex jumping game level map.

[0039] Figure 10 This is a performance comparison chart of the reinforcement learning level testing method of the present invention and two other algorithms.

[0040] Figure 11 This is a comparison chart of strategy diversity indicators.

[0041] Figure 12 This is a comparison of the position heat maps of the method of the present invention and the professional manual testing method. DETAILED DESCRIPTION

[0042] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0043] Developing efficient reinforcement learning methods suitable for game development environments can be divided into five technical areas, from bottom to top: optimizing the scheduling efficiency and stability of underlying training servers; optimizing the performance and scalability of the reinforcement learning task framework; optimizing the reinforcement learning algorithm; optimizing the interaction between the model and the environment; and optimizing the AI ​​application pipeline. This embodiment of the present invention focuses on the latter three dimensions to achieve efficient application of deep reinforcement learning in game testing during the development phase.

[0044] See Figure 1 The embodiment of the present invention provides a reinforcement learning level testing method for the game development stage, including: performing diversity-driven reinforcement learning during the game level testing process, wherein the diversity-driven reinforcement learning algorithm is provided with a task target reward r for the interactive environment. ex and the diversity reward r generated by the diversity evaluation of the current strategy in , where when the diversity of the current strategy is greater than or equal to the threshold δ in And the mission target reward r ex Less than the threshold δ ex When the learning goal is to focus on obtaining more task target rewards r ex; When the diversity of the current strategy is less than the threshold δ in When , the learning goal is to focus on obtaining more diversity rewards r in , to encourage the agent to choose a strategy that it has not taken before.

[0045] In a preferred embodiment, when the diversity of the current strategy is greater than or equal to the threshold δ in And the mission target reward r ex Greater than or equal to the threshold δ ex At any time, the total reward is determined by the task target reward r ex and diversity rewards in The weight of their contribution to the overall reward is determined jointly, and the weight of their contribution to the overall reward is adjusted through hyperparameters.

[0046] The present invention's reinforcement learning level testing method for the game development phase uses a diversity-driven reinforcement learning approach to optimize the reinforcement learning training process and integrates an innovative test reward mechanism into agent behavior pattern training, improving the targeted nature of training objectives and better aligning them with the goals of game testing. The present invention's diversity-driven reinforcement learning reward function design method improves the coverage of game environment exploration and the validity of the generated data when using reinforcement learning algorithms for game testing, better meeting the testing needs of industrialized game development pipelines.

[0047] Specific embodiments of the present invention are further described below.

[0048] In terms of reinforcement learning algorithms, we used and optimized diversity-driven reinforcement learning methods to make them more in line with the needs of game testing.

[0049] Diversity-driven reinforcement learning algorithms, a new reinforcement learning method proposed in recent years, aim to find agents that can complete tasks with the highest reward while using different strategies than the previous ones. When an agent adopts a new strategy that differs significantly from its previous strategy, it receives more rewards; conversely, if it adopts a strategy similar to its previous strategy, it is penalized. Diversity-driven reinforcement learning algorithms can significantly reduce the risk of agents being trapped in local optima and increase the efficiency of exploring complex, multi-solution environments. During training, new strategies are continuously explored, generating new and effective data.

[0050] The optimization of the diversity-driven reinforcement learning algorithm in the embodiments of the present invention includes the following two aspects:

[0051] 1. Optimization of reward function based on reward switching. In the algorithm of the present invention, the optimization objective of the model consists of two parts. The first part is the task target reward r for the interactive environment. ex , this part is similar to other algorithms; the second part is the reward r generated by the diversity evaluation of the current strategyin The diversity of the current strategy is greater than the threshold δ in When , the main reward comes from external rewards; when the diversity of the current strategy is less than the threshold, the external reward will no longer work, and the algorithm will tend to choose strategies that have not been taken before; at the same time, the reward obtained in the current interactive environment is lower than the threshold δ ex When , the diversity reward will not take effect, and the algorithm will tend to obtain more environmental interaction rewards.

[0052] After giving the hyperparameter α used to adjust the proportion of the two parts of the reward, the overall calculation formula of the reward function is:

[0053] R=(1-CMP(r in , δ in )+αCMP(r ex , δ ex ))r in +CMP(r in , δ in )r ex

[0054]

[0055] For r in The diversity evaluation method adopted by the algorithm of the present invention is: for the current strategy π, randomly sample n behaviors in the entire behavior trajectory and combine them into a behavior vector b n (π), and compare it with the k closest behaviors π in the stored strategy library π ′ For comparison, calculate the Euclidean distance as a reward, that is:

[0056]

[0057] The overall framework of the algorithm is shown below. In practical applications, K is generally not more than 5, l is generally not more than 30, and T ′ Generally not less than 5, the behavior sampling method b(π) can adopt different strategies according to different situations. Generally, n behaviors or positions are randomly sampled in the entire behavior trajectory of the strategy to form a behavior vector b representing the strategy. n (π).

[0058]

[0059] In addition, during model training, a group of reinforcement learning models can be maintained to form a strategy library π instead of a single model. In each round of training, the k models π with the largest strategy differences are selected for training.

[0060] 2. Mission Target Rewards based on the Unified Mounted Reward Module exEvaluation function. In actual application, the target reward from the environment is a sparse reward configured by the game designer when designing the level process. After configuration, all rewards from the environment are uniformly processed by the reinforcement learning module. The corresponding reward module is turned on / off according to the agent's exploration progress of the current level, and the reward value of this node is automatically calculated according to the pre-set reward trigger condition. Compared with the global reward mechanism, it is more in line with the requirements of level QA testing to complete a series of sub-goals in a certain topological order. At the same time, the uniformly configured sparse reward can facilitate the estimation of two threshold hyperparameters δ in and δ ex , reducing the pressure of parameter adjustment.

[0061] Regarding the interaction between the model and the environment, ML-Agent, a reinforcement learning environment provided by the Unity engine, abstracts and encapsulates the test metrics commonly used in level testing during game development. A set of quickly configurable reward and data processing modules has been designed. Simply attaching the script to the corresponding Actor and performing preliminary configuration automatically sets the reward function and collects and processes data.

[0062] Specifically, the following modules are designed:

[0063] (1) Reward Module

[0064] In order to implement a universal reward system, we abstracted the common interaction mechanisms and gameplay elements in game testing and divided them into three types: passing, destruction, and target. These are triggered when overlap occurs, when components are removed, and when collision / interaction occurs. The mapping relationship of the reward detector is as follows: Figure 2 shown.

[0065] In specific applications, the management component of each level mounts the global timing and target class reward detectors, and the rewards are set as continuous rewards; the prefabricated bodies of the gameplay elements that appear in the level are all mounted with corresponding reward detectors, and the rewards are set as one-time rewards, corresponding to the sub-goals in the level process.

[0066] Reward detectors that require preconditions can set these preconditions to ensure the correct topology of the level flow. All reward detectors can set reference reward values. When obtaining the reward for the current step during model training, the triggerable rewards will be broadcast and uniformly balanced and standardized to prevent problems such as some rewards being set too large or too small.

[0067] Figure 3 The workflow of the reward module of the embodiment of the present invention is exemplarily shown.

[0068] like Figure 3As shown in the example, in a level requiring three objectives, the testing process involves sequentially completing the three phases of the objectives: collecting two coins, exchanging them for a key, and then opening a gate. The level designer then attached reward detector scripts to the corresponding actors, creating Phase 1 Reward Detector 1: Collect Coin 1, Phase 1 Reward Detector 2: Collect Coin 2, Phase 2 Reward Detector 1: Exchange for Key 1, and Phase 3 Reward Detector 1: Open Gate 1. Furthermore, consistent with the level logic, the pre-trigger condition for Phase 2 Reward Detector 1 is set to trigger both reward detectors in Phase 1, and the pre-trigger condition for Phase 3 Reward Detector 1 is set to trigger the reward detector in Phase 2. Furthermore, the level also contains a globally retriggerable interactive object, attached to Global Reward Detector 1, with no pre-trigger condition. This completes the reward module configuration. During actual training, when Phase 2 Reward Detector 1 is triggered, the reward given to the agent is the sum of the base reward configured for the Phase 2 Reward Detector 1 script and the rewards of all pre-trigger scripts (Phase 1 Reward Detectors 1 and 2). Such a reward module not only makes it easier for designers to configure, but also ensures that the reward for completing the level goal is higher than the reward for interacting with other objects in the scene, converging the agent's exploration behavior on completing the goal.

[0069] (2) Data collection module

[0070] The data collection module is relatively simple, primarily collecting input and position data generated when the reinforcement learning agent runs the intelligent body, and verifying the rationality of the data. It locates vulnerabilities in unreasonable data and directly outputs bug reports; reasonable data is stored locally to facilitate unified processing during data analysis. Data rationality verification is achieved by configuring monitoring items and thresholds in the monitoring script. This can be used in functional testing, performance testing, and user experience testing to monitor non-blocking bugs such as characters exceeding map ranges, exceeding speed limits, and frame rate drops below thresholds.

[0071] During the specific configuration, mount the input record and trajectory record on the Actor corresponding to the character, and mount the BUG monitoring script for the values ​​you want to monitor in the scene.

[0072] Figure 4 This example illustrates the workflow of the data module in the reinforcement learning level testing method of the present invention. The data recorder monitors the actions generated by the character controller, the bug detector verifies whether the configuration items in the monitoring script exceed the threshold, and the trajectory recorder records the agent's position.

[0073] (3) Data processing module

[0074] This work implements visualizations such as location heatmaps for intuitive analysis of data generated during level exploration during training. The location heatmap is implemented by collecting player location data, mapping all locations in the Unity environment, and rendering them to a Texture2D for output. The following figure shows the output location heatmap overlaid on the corresponding game level.

[0075] At the same time, the trained model can be directly imported into the game through Unity's ML-Agent plug-in for replay testing to check the performance of the intelligent agent in the environment.

[0076] Figure 5 The training algorithm architecture of the reinforcement learning level testing method is exemplified.

[0077] like Figure 5 As shown, the training algorithm architecture of the reinforcement learning level testing method of the present invention is as follows: The ML-Agent module provided by Unity is loaded into a Python backend environment. This module includes a socket transceiver, environmental control, agent, and decision logic, but uses the diversity-driven reinforcement learning model training algorithm of the present invention. The reward module and data module designed in the present invention are configured in the Unity environment, and the character controller and environmental sensor provided by ML-Agent are used. After configuration is complete, the reinforcement learning model can be trained.

[0078] In summary, the reinforcement learning level testing method of the present invention realizes effective testing and optimization of game levels through the collaborative work of a series of modules, which helps to improve the quality of the game and the player experience.

[0079] From the perspective of the AI ​​application pipeline, the burden of designing the reinforcement learning reward function is naturally transferred to the level designer's normal level design workflow, requiring less additional configuration effort. The application process of the entire method is shown in the figure below. When designing the level flow, the level designer needs to mount the corresponding reward, data collection, and analysis scripts under the actor corresponding to the script logic. The subsequent testing process can be carried out automatically until the end of the testing process for model review and data analysis, at which point the level designer needs to re-enter the interface.

[0080] For example, a level designer implemented a simple level in the Unity engine. Players must collect a key and then use it to unlock a door. After preliminary process design, the designer needs to attach the character controller script and the environment sensor script to the player's corresponding actor. They also attach the phased collection and pass reward detector scripts to the key and door actors, setting the triggering order. They also attach the input and trajectory recording scripts and the bug monitoring script to any location. Afterwards, they select the reinforcement learning model and start training. Once the model has converged, they will obtain the test results.

[0081] In the above embodiment, from the perspective of optimizing the interaction between the model and the environment, this method preferably uses the encapsulation and unified processing of the reward function to accelerate the reward configuration in the R&D environment. As an alternative, it is also possible to accelerate by optimizing the agent's perception of the environment;

[0082] In the above-mentioned embodiment, from the perspective of AI application pipeline optimization, this method processes the data generated by the reinforcement learning agent's exploration within the level through location heatmap analysis, operation frequency statistics, and trajectory analysis to support subsequent functions such as bug detection and level difficulty analysis. There are also various alternative methods for processing the agent-generated data, such as spectrum analysis. Furthermore, this method is preferably developed based on the ML-Agent environment, a basic reinforcement learning plugin provided by the Unity engine. Pipeline optimization will also vary significantly after changing the game engine and system architecture.

[0083] Figure 6 The implementation process of the reinforcement learning level testing method is shown as an example. Figure 6 As shown, the process includes: test preparation, model training, and test result output. During the preparation phase, level designers complete simple configuration of multiple modules such as rewards and data while building the level. During the model training phase, the model is trained locally or in the cloud using the algorithm provided by this invention. During the test result output phase, the trained reinforcement learning model is imported into the game environment for replay testing, and data analysis and bug detection results are output.

[0084] Experimental verification

[0085] The diversity-driven reinforcement learning testing method constructed by this invention was compared with manual testing methods, random walk methods, and previous reinforcement learning automated testing methods. The results showed that in a given level process, the method of this invention outperformed previous reinforcement learning testing methods in terms of bug detection and behavior coverage, and was not much different from the average performance of manual testing methods. The experimental process is as follows:

[0086] In the Unity engine, we built two 2D platform jumping game levels of varying difficulty as experimental platforms for testing and comparison. We also used the "Space Invaders" game in the OpenAI GYM environment, a commonly used reinforcement learning environment, as an auxiliary experimental platform.

[0087] For both reinforcement learning methods, the step size for each round of the game was set to 1000 steps. After the model training reached convergence (approximately 2 million times), the model was run for another 1000 rounds and the resulting data was analyzed, focusing on bug triggering and the completion of level objectives. Each reinforcement learning method was implemented based on the Probability Proof-of-Play (PPO) algorithm. Three sets of models were independently trained with different random seeds, and the statistical results were averaged. Training was performed using a server cluster, with the game environment deployed to the server via a Docker virtual machine and trained remotely along with the model.

[0088] In the reinforcement learning method, the observation space used is shown in Table 1 below.

[0089] Table 1

[0090]

[0091] Among them, after collecting the observation space data, the observation features of different dimensions were normalized.

[0092] In the reinforcement learning method, the action space used is shown in Table 2 below, where all actions are discrete actions and use one-hot encoding.

[0093] Table 2

[0094]

[0095] For the random walk algorithm, the step length of each round of the game is set to 1000 steps, and it is repeated for 1000 rounds. The data statistics method is the same as the reinforcement learning method.

[0096] For the manual testing control group, we recruited five players of varying skill levels, three game designers, and three professional test engineers. After explaining the game's rules and objectives, we tested whether the game encountered any issues during play. A brief interview was conducted afterward. Each round of the game remained 1,000 steps long, and each tester was given 15 minutes to complete. We also collected statistics on the time it took to discover bugs and complete level objectives, and the average of the results was used.

[0097] In the Space Invaders environment, e.g. Figure 7As shown, the player has only one goal: to shoot down all enemy aircraft while dodging bullets. There are two pre-installed bugs in the game: the player's coordinates can exceed the map limit; and the game ends when all monsters in the leftmost column die.

[0098] For both reinforcement learning methods, the observation space was the grayscaled game screen, and the action space was one-hot encoded. For the method of the present invention, the reward function was set as the Atari score plus the strategy diversity score; for the reinforcement learning method used as a control, the reward function was set as the Atari score plus the curiosity reward. To detect these two types of bugs, both reinforcement learning models incorporated the bug detection script of the present method, which monitored the aircraft coordinates and the remaining step length, respectively.

[0099] The training time of the experimental model and part of the manual testing time are shown in Table 3 below.

[0100] Table 3

[0101]

[0102] In a simple jumping game level, such as Figure 8 As shown, the player has two objectives to complete in sequence: collecting a key and then using the key to open the gate. There are two pre-installed bugs in the game: the player can open the gate without collecting the key; and after the player collects the key, the collision of the trap disappears, causing the player to fall out of the trap and out of the map.

[0103] In complex jumping game levels, such as Figure 9 As shown, the player has three objectives to complete in sequence: collecting three gold coins, using the three coins to purchase a key, and then using the key to open the gate. Three bugs are embedded in the game: players receive two keys when purchasing; after purchasing the key, the collision of the trap disappears, causing the player to fall out of the trap and out of the map; and the game frame rate drops when the gate is opened (using a script to simulate frame drops).

[0104] For the two reinforcement learning methods, the observation space is the grayscale game screen, the 8-direction ray detection results of the character's current position, the number of props held, and the number of level goals completed. The action space uses one-hot encoding to list the possible operations, where interactions with keys and doors are all performed automatically. For the method of the present invention, the reward function is set to the stage goal (get the key / open the door) score plus the strategy diversity score; for the previous reinforcement learning method, the reward function is set to the stage goal score plus the curiosity reward, the observation space is the grayscale game screen, and the action space uses one-hot encoding. Specifically, the reward value given by the stage goal is the remaining step length of the current round.

[0105] The experimental results are summarized in Tables 4 and 5 below.

[0106] Table 4

[0107]

[0108] Note: The interruption of the level process caused by the injection of BUG is regarded as the achievement of the level goal

[0109] Table 5

[0110]

[0111] Figure 10 The performance comparison between the reinforcement learning level testing method of the present invention and the other two algorithms is shown.

[0112] As can be seen, as the level progresses and sub-goals change, traditional methods struggle to complete subsequent levels. In particular, curiosity-based reinforcement learning methods, due to smaller environmental changes and smaller errors in predicting subsequent states, generate less intrinsic curiosity reward after goal updates, making it difficult to adapt to changing goals. Diversity-driven reinforcement learning methods, on the other hand, can learn and explore more efficiently in longer, sequential levels thanks to the state updates brought about by sub-goal updates.

[0113] As a control group, three groups of data from manual testing are shown in Table 6 below.

[0114] Table 6

[0115]

[0116] It can be seen that the effect of this method on BUG monitoring surpassed that of the test group of players and designers, and achieved a level similar to that of the test engineer group.

[0117] At the same time, we collected location information from the manual test group during the test. To compare the strategy coverage of the method of the present invention with that of the manual test method, we defined a strategy diversity score based on the difference in game trajectories during each round of testing. For a strategy group Pi that underwent n rounds of testing, its strategy diversity index was defined as:

[0118]

[0119] Where Φ(π u ) represents the strategy π used in the i-th round of testing i The vector of all the positions reached in sequence. Strategy diversity index comparison Figure 11 shown.

[0120] At the same time, the position heat map of the method of the present invention and the professional manual test method was produced, and the comparison results are as follows: Figure 11 As shown (the upper part of the figure is the method of the present invention, and the lower part is the manual testing method).

[0121] It can be seen that for the process test of complex jumping levels, except for a small number of aerial trajectories with certain differences, in key areas, the method of the present invention achieved a behavior coverage rate similar to that of professional manual methods.

[0122] In summary, in view of the fact that existing game testing requires a large amount of professional knowledge in the testing field and complex parameter configuration, as well as the current reinforcement learning level automation testing method does not match the rapid iteration requirements of the game development stage and the high cost of the level testing method, the embodiment of the present invention proposes a game level testing method based on diversity reinforcement learning, which integrates the innovative test reward mechanism into the intelligent agent behavior pattern training, improves the pertinence of the training objectives, and is more in line with the goals of game testing. The diversity-driven reinforcement learning reward function design method improves the coverage of the game environment exploration and the validity of the generated data when using the reinforcement learning algorithm for game testing. Various test indicators are designed based on the industrialized game production pipeline to meet the various special testing requirements of the game development stage, and have a higher coverage of various test requirements in the game development stage. The present invention is simple to use and has a short training time. Compared with other reinforcement learning methods, it can effectively save time and cost.

[0123] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0124] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0125] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0126] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0127] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0128] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0130] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0131] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0132] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0133] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0134] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0135] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A reinforcement learning level testing method for the game development stage, characterized by: include: Diversity-driven reinforcement learning is performed during the game level testing process. The diversity-driven reinforcement learning algorithm sets task target rewards for the interactive environment. and the diversity reward generated by the diversity evaluation of the current strategy , where when the diversity of the current strategy is greater than or equal to the threshold And the mission goal reward Less than threshold When learning, focus on obtaining more task target rewards ; When the diversity of the current strategy is less than the threshold When learning, focus on obtaining more diverse rewards , to encourage the agent to choose a strategy that it has not taken before; When the diversity of the current strategy is greater than or equal to the threshold And the mission goal reward Greater than or equal to the threshold When the total reward is determined by the mission target reward and diversity awards The weight of their contribution to the overall reward is determined jointly, and adjusted through hyperparameters; It also includes: forming a unified mount point for centralized management of reward modules, and forming a sparse reward script that can be mounted on any object under unified management, so that when the level designer designs the level process, the sparse reward script can be mounted on the level interactive object to configure the corresponding task target reward; during training, the unified mount point provides unified management of all mounted reward scripts, thereby simplifying the threshold and Estimation of and design of reward function.

2. The reinforcement learning level testing method for the game development stage according to claim 1, characterized in that: Overall reward function The calculation formula is: ； ； in, Is a conditional judgment function. Corresponding to or or , is a hyperparameter.

3. The reinforcement learning level testing method for the game development stage according to any one of claims 1 to 2, characterized in that: For diversity awards , for the current strategy , randomly sample the entire behavior trajectory Behaviors are combined into behavior vectors and compare it with the stored policy library in The closest behavior Compare and determine the reward based on the calculation of Euclidean distance, that is: 。 4. The reinforcement learning level testing method for the game development stage according to any one of claims 1 to 2, characterized in that: The corresponding reward module is dynamically turned on or off according to the agent's exploration progress of the current level to guide the agent to complete a series of sub-goals in the topological order of the level design.

5. The reinforcement learning level testing method for the game development stage according to any one of claims 1 to 2, characterized in that: Also includes: The various interactive mechanisms and gameplay elements used in level testing during the game development phase are abstracted and encapsulated to form a set of standardized test indicators. This is combined with the plugins provided by the game engine to set up a quickly configurable reward module and data processing module. By mounting the script on the corresponding Actor and performing preliminary configuration, the reward function is automatically set and data is collected and processed.

6. The reinforcement learning level testing method for the game development stage according to any one of claims 1 to 2, characterized in that: For the reward module, we define different reward types and set corresponding trigger conditions to achieve automated management of rewards, and balance and standardize rewards; The data collection module collects the input and location data generated when the reinforcement learning agent runs the intelligent body, and configures monitoring scripts and thresholds to verify the rationality of the data; the data processing module processes and analyzes the collected data to realize the location heat map and provide visualization effects.

7. The reinforcement learning level testing method for the game development stage according to any one of claims 1 to 2, characterized in that: Also includes: When designing the level process, the corresponding script is mounted in the appropriate location, and the subsequent testing process is carried out automatically. The level designer is not required to re-intervene until the test process is completed and the model is reviewed and data analyzed.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the reinforcement learning level testing method for the game development stage as described in any one of claims 1 to 7.

Images