Pixel-based automated testing of navigable simulation environments

Abstract

A computing system for pixel-based automated testing for testing a navigable simulation environment includes one or more processors configured to execute an application test program during a runtime inference phase. The application test program includes a machine learning model trained to detect errors in the application under test. A screen image of the simulated environment is processed by an object detection module to determine whether there is a key object in the screen image. If there is a key object in the screen image, the test program is applied to execute an object investigation module to generate an investigation input to investigate the key object. If there is no key object in the screen image, the application test program executes an environment exploration module to generate an environment exploration action that is to be provided to the application to be tested as a simulated user input.

Description

Background Art

[0001] Developing computer programs for virtual environments (such as virtual reality environments, computer game environments, simulator environments, etc.) that are navigated by users requires extensive testing to ensure that everything is free of bugs, glitches, and other errors. Testing such programs for quality assurance typically requires human testers and / or manually written scripts for testing, and relies on internal state information of the program. As technological advances in program development have led to increases in the size and complexity of these programs and their virtual environments, thoroughly testing these programs has become increasingly complex and time-consuming. Summary of the invention

[0002] In view of these problems, a computing system for pixel-based automated testing is provided, and the automated testing is used to test a navigable simulated environment generated by a tested application. The computing system has server-side and client-side aspects. According to one aspect, the computer system includes one or more processors, and the one or more processors are configured to use a portion of an associated memory to execute instructions to implement an application test program in a runtime reasoning phase. The application test program includes a machine learning model trained to predict errors in the tested application. The application test program is configured to receive a screen image of a navigable simulated environment generated by the tested application, and to process the screen image using an object detection module to determine whether there is a key object in the screen image. If it is determined that there is a key object in the screen image, the application test program is configured to execute an object investigation module, which receives an image of the detected key object and generates an investigation input for investigating the key object. If it is not determined that there is a key object in the screen image, the application test program is configured to execute an environment exploration module, which receives a screen image as input, and processes the screen image to generate an environment exploration action, which will be provided to the application to be tested as a simulated user input.

[0003] In some configurations, the object detection module includes an object detection neural network for detecting key objects in the application under test. The object detection neural network is trained during an initial training phase using a first training data set derived from common objects in the context data set, and the object detection neural network is subsequently trained using a second training data set that includes annotated screen images of key objects in the application under test.

[0004] In some configurations, the object investigation module includes an object investigation neural network that is trained using investigation input from human testers investigating key objects during human game testing sessions.

[0005] In some configurations, the environment exploration module includes a target neural network and a predictor neural network. During the exploration phase, a screen image is input to the target neural network and the predictor neural network; and each of the target neural network and the predictor neural network maps the screen image to a corresponding embedding vector. The target neural network is randomly initialized and fixed, and the output embedding vector of the target neural network during the exploration phase is set to the true value for the predictor neural network. The predictor neural network outputs a predicted embedding vector and is trained using a distillation loss function to minimize the prediction error between the output embedding vector of the target network and the predicted embedding vector. The environment exploration module also includes an exploration policy neural network, which is configured to predict an environment exploration action that will produce the maximum expected reward in response to receiving the screen image as input.

[0006] This summary is provided to introduce a selection of concepts in simplified form, which will be further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is an overall schematic diagram of a computing system for pixel-based automated testing of a navigable simulated environment generated by an application under test according to an embodiment of the present disclosure.

[0008] Figure 2 is used Figure 1 Diagram of the decision flow for pixel-based automated testing of computing systems.

[0009] Figure 3A yes Figure 1 Schematic diagram of the object detection neural network of the computing system.

[0010] Figure 3B and Figure 3C During the initial training phase Figure 1 Schematic diagram of the object detection neural network of the computing system.

[0011] Figure 3D During the inference time phase Figure 1 Schematic diagram of the object detection neural network of the computing system.

[0012] Figure 4A During the initial training phase Figure 1 Schematic diagram of a neural network for object investigation of computing systems.

[0013] Figure 4B During the inference time phase Figure 1 Schematic diagram of a neural network for object investigation of computing systems.

[0014] Figure 5 During the exploration phase Figure 1 Schematic diagram of the curiosity-driven environment exploration module of the computing system.

[0015] FIG. 6A to FIG. 6E Shows the use of Figure 1 Exploration of navigable simulated environments by computational systems.

[0016] 7A to 7E Shows the use of Figure 1 A computational system to investigate key objects in a navigable simulated environment.

[0017] Figure 8 A flowchart of a method for pixel-based automated testing of a navigable simulation environment generated by an application under test according to an example of the present disclosure is shown.

[0018] Fig. 9 It shows that it can be achieved Figure 1 Schematic diagram of an example computing environment of a computer system.

[0019] Fig.10 Shown by Figure 1 Pseudocode of the algorithm implemented by the system. DETAILED DESCRIPTION

[0020] Testing a computer program in a virtual environment (such as a virtual reality environment, a computer game environment, a simulator environment, etc.) navigated by a user involves exploration of the environment and identification of objects within the environment and investigation of the identified objects. Inadequate testing may result in the release of a computer program with vulnerabilities, failures, and other errors, which may result in substandard program performance, a frustrating user experience, emergency recalls or updates of computer programs, and consumption of computer and human resources to solve problems. Testing by human users can minimize the presence of vulnerabilities, failures, and other errors in computer programs; however, human testing is typically expensive and time-consuming. In addition, as computer programs become larger and more complex, it becomes less feasible to fully test using human users.

[0021] Automated computer program testing, such as using deep reinforcement learning (DRL) algorithms, can be performed as an alternative to human testing. However, testing using DRL algorithms typically relies on internal state information, which requires deep integration with the source code and therefore requires access to the source code. This deep integration with the source code limits the application of the testing algorithm to computer programs that only have available source code. In addition, even when accessible, the source code changes frequently during computer program development, making it challenging for developers to maintain deep integration of the testing algorithm.

[0022] like Figure 1 As schematically shown in , in order to solve the problems identified above, a computing system 10 for pixel-based automated testing of a navigable simulated environment generated by an application under test is provided. The computing system 10 is shown to include a first computing device 12 and a second computing device 18, the first computing device 12 including one or more processors 14 and a memory 16, and the second computing device 18 including one or more processors 20 and a memory 22, wherein the first computing device 12 and the second computing device 18 communicate with each other via a network 24. The illustrated embodiments are exemplary in nature, and other configurations are possible. In the following Figure 1 In the description of the present invention, the first computing device will be described as server computing device 12 and the second computing device will be described as test computing device 18, and the corresponding functions performed at each device will be described. It should be understood that in other configurations, the first computing device may be a computing device other than server computing device 12. In some configurations, computing system 10 may include a single computing device that performs significant functions of both server computing device 12 and test computing device 18. In other alternative configurations, functions described as being performed at test computing device 18 may alternatively be performed at server computing device 12, and vice versa.

[0023] continue Figure 1 For example, the server computing device 12 can take the form of an automated testing server hosted in a data center of an application testing platform operator. Alternatively, the server computing device 12 can be configured by the application testing platform operator to be hosted at a client-specified data center under the control of the client for reasons such as performance, security, privacy, and / or export compliance. As described above, the server computing device 10 includes one or more processors 12 and associated memory 14. The one or more processors 12 are configured to use portions of the associated memory 14 to execute instructions to execute an application testing program 26 including an artificial intelligence (AI) model during a reasoning time phase that explores a navigable simulated environment, detects and investigates objects within the navigable simulated environment, and detects errors in an application under test (AUT) 28 that generates a virtual environment.

[0024] In the embodiments described herein, the AI ​​model is a machine learning (ML) model, referred to as a virtual environment inspection ML model 30, and the AUT 28 is a computer game application. However, it should be understood that the ML model can be implemented as a pixel-based test model for testing any type of computer application compatible with pixel-based automated testing. Similarly, the AUT 28 can be any type of computer application with a navigable simulation environment suitable for pixel-based automated testing, such as a computer game, a virtual reality program, or a three-dimensional simulation program (e.g., a flight simulator, a mechanical simulator, etc.). Additionally, when the AUT 28 is configured as a game, it can be configured as a single-player or multi-player game, and for simplicity, the computer game application discussed herein is described as a single-player game.

[0025] The AUT 28 may include program logic 32 configured to be executed by one or more processors 20 using associated memory 22 to implement the functionality of the AUT 28 described herein. A navigable simulated environment 34 of the AUT 28 may be displayed in a user interface (UI) 36 on a display 38 of the test computing device 18. For example, the navigable simulated environment 34 may be three-dimensional or two-dimensional, and may be displayed from the perspective of a first-person user or from another perspective. During testing, the application test program 26 may receive from the AUT 28 a screen image 42 (i.e., a screenshot) of the navigable simulated environment 34 (e.g., a game space in the case of a computer game) calculated by the program logic 32 based on simulated user input 54. In some embodiments, the application test program 26 may send a request 40 for the screen image 42 to the AUT 28. Notably, the application test program 26 typically only receives the screen image 42 as input from the AUT 28 for controlling the application test program 26 to explore the navigable simulated environment 34, and is therefore configured to execute independently of the internal state of the AUT 28. It should be appreciated that the screen image 42 may be processed to produce pixel-by-pixel rendered image data by the application testing program 26 .

[0026] The screen image 42 is processed using the object detection module 44 included in the virtual environment inspection ML model 30. FIG. 3A to FIG. 3DAs described in detail, the object detection module 44 includes an anchor-based object detection neural network (NN) 46, which is configured to determine whether a key object is present in the screen image 42. If a key object is present, the object detection NN 46 determines the type and location of the key object, and the virtual environment inspection ML model 30 executes an object investigation module 48, which receives an image of the detected key object. The object investigation module 48 includes an object investigation NN 50, which is configured to investigate the key object by inputting a series of investigation inputs 51 predicted to be made in response to the screen image 42. The object investigation NN 50 is trained using a training data set that includes screen image pairs 42 and true value investigation inputs made by human software testers during testing. Therefore, the object investigation NN learns to imitate a human tester based on the screen image data. The investigation input can be, for example, a series of keystrokes, game controller inputs, mouse clicks, touch screen touches, touch pad touches, etc.

[0027] The simulated input generation module 52 receives one or more predicted survey inputs 51 from the object survey module 48 in response to its processing of the screen image 42. The simulated input generation module 52 generates simulated user inputs 54 based on the one or more predicted survey inputs 51 and sends the simulated user inputs 54 to the test computing device 18. The simulated user inputs can be, for example, application program interface (API) calls, Bluetooth commands, or commands sent via another software or hardware interface that simulate actual user inputs from a real user input device, even if the real user input device is not connected during testing. In this way, the simulated user inputs 54 can provide commands to the AUT 28 that implement the predicted survey inputs 51. The simulated user inputs 54 cause the program logic 32 of the AUT 28 to update the navigable simulated environment 34 of the UI 36, which causes the updated screen image 42 to be output. The simulated user inputs 54 can include actions that a player can take in a computer game, such as move forward, move backward, move left, move right, turn left, turn right, and jump. Simulated user input 54 may be in the form of keyboard commands, game controller commands, mouse clicks, touch screen input, touch pad input, etc., for example, provided by a hardware or software interface as described above.

[0028] When it is uncertain whether a key object exists in the screen image 42, or when the object investigation is completed, the virtual environment inspection ML model 30 is configured to execute the environment exploration module 56. The environment exploration module 56 includes three environment exploration NNs 58A, 58B, 58C, which are configured to process the screen image 42 to predict whether the screenshot 42 has been seen before, and predict environment exploration actions. The environment exploration module 56 includes a reward function that rewards actions that cause unfamiliar screen images to be displayed. Therefore, the environment exploration module 56 is motivated to predict environment exploration actions that are used to explore novel areas in the game space that have not been frequently explored during testing. These predicted environment exploration actions can be received by the simulated input generation module 52, which generates at least one exploration action to be provided to the AUT28 as simulated user input 54. In the UI 36 of the AUT 28, the simulated user input 54 is sent to the AUT 28 and causes the environment 34 to be updated by the program logic 32.

[0029] If an error is found in the simulated navigable environment while investigating key objects and / or exploring the navigable simulated environment, the object investigation module 48 and / or the environment exploration module 56 can transmit a record of the error to be entered into the test report 60, including the location in the environment, any key object involved, and the type of error detected. The test report 60 can be transmitted to the test computing device 18 as shown in the figure, or transmitted to another storage location. If no error is detected, the application test program 26 can be configured to send a record of no error detected for the location in the environment to be included in the test report 60. Alternatively, if no error is detected, the application test program 26 can not send any record to be included in the test report for the location, so that the test report only contains errors. The test report 60 can be reviewed by the game developer during testing to determine the location and key objects with errors in the navigable environment, and the corresponding areas of the game source code that need attention and repair.

[0030] Figure 2is a decision flow chart for pixel-based automated testing using an application tester 26. The object detection module 44, the object investigation module 48, and the environment exploration module 56 together comprise an integrated system for testing the application under test. As shown in the decision flow, at each time step t, the object detection module 44 detects whether a key object is present in the screen image 42. If not, the environment exploration module 56 will be executed to explore the navigable simulated environment 34 through simulated user input 54 that simulates user actions. If a key object is detected, the object investigation module 48 is executed, and the simulated user input 54 according to a series of actions is executed to investigate the key object. When the investigation is complete, the exploration of the navigable simulated environment will continue via the environment exploration module 56 until the key object is found.

[0031] An important aspect of game testing is interacting with key objects in the game space, because some hidden and hard-to-find vulnerabilities can only be detected via player interaction with key objects. However, identifying and labeling key objects in the game space can present challenges. In order to detect key objects and better expose potential errors associated with key object interactions, the application tester 26 includes an object detection NN 46 and an object investigation neural NN 50.

[0032] A schematic diagram of anchor-based object detection NN 46 is shown in Figure 3A As shown. As shown, object detection NN 46 is configured as a region-based fast convolutional neural network (R-CNN), which includes a feature extractor 62 and a box predictor 64, as shown in dash-point and dash-point-point, respectively. Feature extractor 62 includes one or more shared convolutional layers (e.g., ResNet) 66, a region proposal network (RPN) 68, and a region of interest (RoI) feature extractor 70. Box predictor 64 includes an object classifier (e.g., SoftMax) 72 for classifying detected key objects into categories, and a box regressor 74 for predicting bounding box coordinates for key objects. When an image such as screen image 42 is input to object detection NN 46, the image is filtered via shared convolutional layer 66, and two feature maps 76 and 78 are generated. In RPN 68, feature map 78 is processed via both object classifier (e.g., SoftMax) 80 and box regressor 82 to generate a RoI proposal 84 for predicting the position of the key object in screen image 42. The RoI proposals 84 from the RPN 68 are pooled with the fixed feature map 76 in the RoI pooling layer 86. The RoI feature extractor 70 extracts features (i.e., objects) from the predicted region of interest, and a bounding box with anchor points and a predicted classification for each object detected in the screen image 42 is output via the box predictor 64.

[0033] In order to minimize the cost and time typically associated with manually annotating large numbers of screen images, the initial training phase for the object NN 46 is implemented as a two-stage fine-tuning approach for few-shot object recognition. Figure 3B As shown, during a first initial training phase, the object detection NN 46 is trained using a first training dataset 88 derived from the Common Objects in Context (COCO) dataset 90, which is a large-scale object detection, segmentation, and captioning dataset that includes more than 200,000 annotated images and multiple captions for each image. Alternatively, in addition to COCO, another dataset showing common objects and their contextual environments may be used. The object detection NN 46 is then trained using a second training dataset 92 during a second initial training phase, as shown in FIG. Figure 3C The second dataset 92 includes annotated screen images 94 for each type of key object in the AUT 28, and ground truth 96 for the key objects in the corresponding screen images. In the second initial training phase, the feature extractor 62 is fixed, and the features learned from the base categories using the COCO dataset 88 during the first initial training phase are applied to the new categories of key objects in the AUT 28.

[0034] Figure 3D 4 is a schematic diagram of an object detection NN 46 during the inference time phase. As shown and described above, the object detection NN 46 is configured to receive as input the screen image 42 of the navigable simulation environment 34 from the test computing device 18. Figure 4A and Figure 4B In more detail, if a key object is detected in the screen image 42, the object detection NN 46 outputs image data of the identified key object within the bounding box coordinates 98 predicted by the box regressor 74, and the object investigation module 48 is triggered to investigate the key object. Figure 5 In more detail, if no key objects are detected in the screen image 42 , the environment exploration module 56 is triggered to explore the navigable simulated environment 34 .

[0035] Figure 4A28. A schematic diagram of an object investigation NN 50 during the initial training phase is shown in FIG. In automated game testing, it is important that the application tester 26 interacts with key objects in the game space in a human-like manner, and it is expected that different types of objects will be studied in different ways by various players. To this end, the object investigation NN 50 is trained for each type of key object in the AUT 28 using a training data set 100, which includes training pairs 102 of pixel-by-pixel rendered image data of key objects in the navigable simulated environment 34 as input 104 and human tester actions made in response to the image data as true values ​​106. As shown, the object investigation NN 50 can be configured as a convolutional neural network (CNN) having an input layer, one or more convolutional layers, one or more fully connected hidden layers, and an output layer. It should be understood that the object investigation NN 50 can be configured to include one or more convolutional layers, one or more fully connected hidden layers, or both one or more convolutional layers and one or more fully connected hidden layers. The input layer includes a plurality of nodes corresponding to the training phase image data input 104, which includes one or more key objects and associated bounding box coordinates 98 derived from the object detection NN 46. The output layer includes a plurality of output nodes corresponding to the ground truth output 106, which indicates the human actions performed by the human tester of the AUT 28 when interacting with the key objects. The nodes in each layer are linked by weighted associations, and the object investigation NN 50 is configured to adjust the internal weights according to the loss function during training to enhance the image data of one or more key objects and the associated bounding box coordinates 98 via a back-propagation algorithm so as to minimize the behavior cloning loss and increase the accuracy of the output nodes during the inference time phase. Once trained, the object investigation NN 50 is able to interact with the key objects in a human-like manner and investigate the key objects to expose potential vulnerabilities in the AUT 28. It should be understood that the proximity of the key objects to the simulated player can be determined by the size of the bounding box associated with the key objects from the object detection NN 46.

[0036] Figure 4B is a schematic diagram of the subject investigating the NN 50 during the inference time phase. Figure 4B 4 and 5. The object detection NN 46 is included in the figure to illustrate the coordination of the object detection module 44 and the object investigation module 48 in the virtual environment inspection ML model 30 of the application test program 26. As described above, the object detection NN 46 is configured to process the screen image 42 and output image data for the key objects identified within the bounding box coordinates 98. The image data 108 is received as input by the object investigation NN 50, and the object investigation NN 50 outputs at least one predicted human-like investigation action 110. As described above with reference to Figure 1As described, the simulated input generation module 52 receives at least one predicted investigation action 110 from the object investigation module 48, generates simulated user input 54 based on the at least one predicted investigation action 110, and sends the test input 54 to the test computing device 18, where one or more actions are performed via the UI 36 of the AUT 28. Also as described above, if an error is discovered while investigating a key object, the object investigation module 48 can send the error to the AUT 28, where the error is entered into the test report 60.

[0037] If no key objects are detected in the screen image 42, the environment exploration module 56 is triggered to explore the navigable simulated environment 34. In game testing, it is critical that the entire game space is explored, and preferably explored efficiently, avoiding unnecessary revisits of previously explored environments. As described above, the environment exploration module 56 includes three NNs 58A, 58B, 58C that coordinate to drive exploration of novel areas in the navigable simulated environment 34.

[0038] Figure 5 The environment exploration module 56 is shown during the exploration phase. Figure 5 As shown in detail in FIG, the first two neural networks included in the environment exploration module 56 are the target NN 58A and the predictor NN 58B applied using the random network distillation (RND) method, and include a curiosity-driven reward function that rewards the exploration of the unfamiliar screen image 42. Both the target NN 58A and the predictor NN 58B may include three convolutional layers and three fully connected layers. Alternatively, one or both of the target NN 58A and the predictor NN 58B may include more or less than three convolutional layers and three fully connected layers.

[0039] The target NN 58A is a randomly initialized and fixed network, in which the weight values ​​of different connections between layers are randomly assigned to increase the entropy of the target NN 58A, thereby improving the learning efficiency. The weight values ​​are fixed during the testing process.

[0040] During the exploration phase, the training predictor NN 58B is trained based on the screen images seen during the exploration of the navigable simulated environment 34. During the exploration of the navigable simulated environment 34, at each time step, the screen image for that time step is input to the target NN 58A and the predictor NN 58B, which are configured to map each screen image to a corresponding embedding vector. Figure 5As shown, the target NN 58A outputs a true value 112 for the predictor NN 58A (e.g., in the form of a true value embedding vector), and the predictor NN 58A outputs a prediction value 114 (e.g., in the form of a predicted embedding vector) as to whether the screen image has been seen before. In order to train the predictor NN 58B, a prediction loss is calculated. A loss function such as a distillation loss function can be used to calculate the prediction loss. For example, the predictor NN 58B can be trained using a distillation loss function to minimize the prediction error 116 between the output embedding vector of the target network and the predicted embedding vector. In one example, the predictor NN 58B can be trained to minimize the mean square error between the true value 112 and the predicted value 114 as the prediction error 116. It should be understood that if the screen image 42 has been seen before, the prediction error 116 will be low, and the reward 118A from the RND-based curiosity-driven reward function 118 will also be low. Therefore, the environment exploration module 56 is motivated to explore the novel screen image 42 of the navigable simulated environment 34.

[0041] The third neural network included in the environment exploration module 56 is an exploration policy NN 58C, which is trained via a proximal policy optimization (PPO) algorithm to receive rewards 118A and output at least one predicted human-like environment exploration action 120 to explore the navigable simulated environment 34, thereby enabling the capture of novel screen images 42. Once trained, the exploration policy NN 58C is configured to predict an environment exploration action that will maximize the expected reward from the reward function 118 in response to receiving a screen image as input. As described above with reference to Figure 1As described, at least one predicted environment exploration action 120 is received by the simulated input generation module 52. The simulated input generation module 52 generates a simulated user input 54 based on the at least one predicted exploration action 120, and sends the simulated user input 54 to the test computing device 18, where the simulated user input causes the program logic to change the internal state of the AUT 28, which causes the navigable simulated environment 34 displayed in the UI 36 to change. This causes the screen image 42 to be updated at the next time step of the reinforcement learning cycle in which the exploration policy NN 58C is trained. The reward function 118 calculates a reward 118A for the predicted environment exploration action 120 output by the exploration policy NN 58C based on the prediction error 116 of the screen image 42 generated as a result of the environment exploration action 120. When the prediction error 116 is larger, the reward 118A is also larger, so that during the exploration phase, when the exploration strategy NN 58C takes an environment exploration action 120 that makes the screen image 42 less similar to the previous screen image 42 input into the target NN 58A and the predictor NN 58B at the previous time step of the reinforcement learning cycle, a larger reward 118A is provided to the exploration strategy NN 58C. Also as described above, if an error is found while exploring the navigable simulated environment 28, the environment exploration module 56 can send the error to the AUT 28, where the error is entered into the test report 60. If no error is found, a record that no error was found can be included in the test report 60, or no report can be generated. The exploration strategy NN 58C can include four convolutional layers and four fully connected layers. However, it should be understood that the exploration NN strategy 58C can include more or less than four convolutional layers and four fully connected layers.

[0042] As described herein, the object detection module 44, the object investigation module 48, and the environment exploration module 56 comprise an integrated system for testing the application under test. The object detection module 44 detects whether a key object is present in the screen image 42. If not, the environment exploration module 56 will be executed to explore the navigable simulated environment 34. If a key object is detected, the object investigation module 48 will be executed to investigate the key object. When the investigation is complete, the exploration of the navigable simulated environment 34 will continue via the environment exploration module 56 until the key object is found. Object investigation and environment exploration are performed in this manner until the entire game space of the AUT 28 has been covered.

[0043] Fig.10 The pseudo code of the algorithm (Algorithm 1) implemented by the above integrated system is shown. As shown in the figure, the algorithm runs through a while loop from time step t to N, and in each while loop, a key detection detector (e.g., object detection module 44) is used to try to detect the screenshot s for that time stept (e.g., screen image 42). If the bounding box sizes generated by the key detection detectors are all larger than a predetermined threshold size, and the predicted probability of the key detection detector exceeds a predetermined confidence threshold, then the investigation strategy network π is used investigate The predicted object is investigated (e.g., by the object investigation NN 50 of the object investigation module 48) for a predetermined number of time steps L. If the bounding box size is below a predetermined threshold or the probability is below a predetermined confidence threshold, then the exploration policy network π explore (e.g., exploration strategy NN 58C) is called to t Exploration action a is generated when t The exploration action is fed into the AUT as input, and the next screenshot s for the next time step is t+1 The AUT's program logic selects the action a t Then, the RND-based curiosity reward is generated by the reward function (e.g., reward function 118), as described above, when the exploration policy network π explore This reward function rewards the exploration policy network π when the action taken produces a relatively novel screen image compared to other screen images that were frequently encountered before the exploration phase. explore Then, the PPO loss is used to explore the policy network π every M time steps explore The explored predictor networks (e.g., predictor NN 58B) are batch optimized using a distillation loss compared to a fixed target network f (e.g., target network NN 58A). The result is a computationally efficient algorithm that maximizes the exploration of newly encountered areas within a navigable environment, thereby increasing the likelihood of error discovery, which results in a reduction in the number of errors in the release version of the AUT.

[0044] exist FIG. 6A to FIG. 6E An example exploration of a navigable simulation environment by an application testing system is depicted in FIG. FIG. 6A to FIG. 6E is shown from the perspective of the simulated player. Fig. 6A 4, key object 122 has been identified in screen image 42. In the illustrated example, key object 122 is configured as a health pack. However, it should be understood that, for example, key objects may be configured as any number of game elements, such as food, clothing, supplies, armor, weapons, vehicles, currency, tokens, etc. As described above, the proximity of key object 122 may be determined by the anchor points of the bounding box associated with key object 122. Fig. 6A , when the key object appears on the other side of the room in the navigable simulated environment 34, the simulated user input 54 may direct the simulated player to move forward to begin investigating the key object 122, such as Figure 6B shown. Figure 6C The following diagram shows the process of navigating from within the navigable simulation environment 34. Figure 6B The position in is rotated 180 degrees, and Fig.6D Shows from Figure 6B Once the key object 122 is investigated, the environment exploration module 56 will be executed to explore new areas of the navigable simulated environment 34, such as Fig. 6E shown.

[0045] 7A to 7E An investigation of a key object 122 in a navigable simulated environment 34 is depicted. FIG. 6A to FIG. 6E Same, Figures 7A to 7E is shown from the perspective of the simulated player. Fig. 7A In the example shown, the key object 122 is depicted as being in close proximity to the simulated player. Thus, the simulated user input 54 can direct the simulated player to move around the key object 122 so as to inspect the key object 122 from all sides. Accordingly, Figure 7B The key object 122 is shown after the simulated player has moved 90 degrees to the left. Figure 7C shows examining key object 122 by moving an additional 90 to 180 degrees to the left from the starting position of the survey, and Fig.7D The key object 122 is shown when the simulated player moves from the initial survey position to 270 degrees. Fig. 7E At this point, the simulated player returns to the starting position of the investigation, and the investigation is completed. At this point, the application testing program 26 can trigger the environment exploration module 56 to find the novel screen image 42 to be tested.

[0046] Figure 8 A flowchart of a method 800 for pixel-based automated testing of a navigable simulation environment generated by an application under test according to an example of the present disclosure is shown. The method 800 is preferably implemented across a test computer device executing the application under test and a server computer device executing an application test program.

[0047] At step 802, method 800 may include: at one or more processors of a computing system, during an inference time phase, receiving, by an application tester, a screen image of a navigable simulated environment generated by an application under test, the application tester including a machine learning model trained to detect errors in the application under test.

[0048] Continuing from step 802 to step 804, method 800 may include processing the screen image using an object detection module to determine whether a key object is present in the screen image. The object detection module may include an anchor-based object detection neural network for detecting key objects in the application under test. The object detection neural network may be trained during a first initial training phase and a second initial training phase. In the first initial training phase, the object detection neural network may be trained using a first training data set derived from common objects in the context data set. In the second initial training phase, the object detection neural network may be trained using a second training data set, the second training data set including annotated screen images of key objects in the application under test.

[0049] Proceeding from step 804 to step 806, method 800 may include determining whether a key object exists in the screen image. If it is determined that a key object exists in the screen image, then at step 808, method 800 may include executing an object investigation module that receives an image of the detected key object. Proceeding from step 808 to step 810, method 800 may include generating an investigation input to investigate the key object. The object investigation module may include an object investigation neural network that is trained using a training data set that includes pixel-by-pixel rendered image data of a navigable simulated environment (e.g., screen image 42) and human tester input (e.g., investigation input 51) made in response to the image data as truth values.

[0050] If it is not determined that a key object exists in the screen image, then at step 812, method 800 may include executing an environment exploration module that receives the screen image as input and processes the screen image. The environment exploration module may include a target neural network and a predictor neural network, and the target neural network may be randomly initialized and fixed. The method may also include inputting the screen image to the target neural network and the predictor neural network during the exploration phase, and mapping the screen image to a corresponding embedding vector via each of the target neural network and the predictor neural network. The method may also include setting the output embedding vector of the target neural network to a true value for the predictor neural network during the exploration phase, outputting a predicted embedding vector via the predictor neural network, and training the predictor neural network using a distillation loss function to minimize the prediction error between the output embedding vector of the target network and the predicted embedding vector.

[0051] Continuing from step 812 to step 814, method 800 may include generating an environment exploration action to be provided to the application under test as simulated user input. The exploration neural network may be trained via a proximal policy optimization (PPO) algorithm to receive rewards and generate environment exploration actions to explore the navigable simulated environment 34, thereby enabling the capture of novel screen images 42. Therefore, the environment exploration module may include an exploration policy neural network that is configured to predict the environment exploration action that will produce the maximum expected reward in response to receiving the screen image as input. In addition, the method may include executing a reward function to calculate a reward for an environment exploration action taken by the exploration policy neural network based on a prediction error for the screen image produced as a result of the environment exploration action, such that when the prediction error is larger, the reward is also larger, thereby providing a larger reward to the exploration policy neural network when the exploration policy neural network takes an environment exploration action that results in a screen image that is less similar to a previous screen image that has been input into the target neural network and the predictor neural network during the exploration phase.

[0052] Extensive testing during computer program development helps ensure that all computer programs are free of loopholes, faults and other errors. However, technological advances have led to larger and more complex computer programs, and thorough testing thereof has become increasingly expensive, complex and time-consuming. The system and method described herein provide a technical solution for testing advanced computer programs using an application tester, which has a virtual environment (such as a virtual reality environment, a computer game environment, a simulator environment, etc.) navigated by a user, and the application tester includes multiple neural networks across three integrated modules for detecting objects within the computer program, investigating detected objects, and exploring the environment. The application tester only receives screen images as input, with the purpose of predicting exploration actions and object investigation inputs, and is therefore configured to be executed independently of the internal state of the application under test. This configuration enables the application tester to be used with first-party applications and third-party applications, because proprietary API customization is not required, thereby improving the interoperability of the application tester. In addition, the application tester can be applied to almost any computer program, because screen images of different sizes from various programs can be easily resized to a size suitable for the application tester. Furthermore, since the exploration strategy neural network uses random network distillation, it does not need to be trained on any application-specific images and can therefore be applied to screen images from almost any AUT, thereby increasing the breadth of its usability. In addition, computer program developers can update the source code of the application under test during testing without having to maintain deep integration of the test algorithm as the source code changes during program development. Therefore, a wide variety of computer programs can be effectively and comprehensively tested without access to source code and with little manpower investment, thereby reducing the cost, complexity, and duration of testing.

[0053] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0054] Fig. 9 A non-limiting embodiment of a computing system 900 that can implement one or more of the above methods and processes is schematically shown. The computing system 900 is shown in simplified form. The computing system 900 can embody the above-described and Figure 1 1. The computing system 900 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices, and wearable computing devices (such as smart watches and head-mounted augmented reality devices).

[0055] The computing system 900 includes a logic processor 902, a volatile memory 904, and a non-volatile storage device 906. The computing system 900 may optionally include a display subsystem 908, an input subsystem 910, a communication subsystem 912, and / or Fig. 9 Other components not shown.

[0056] Logical processor 902 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, convert the state of one or more components, implement technical effects, or otherwise achieve desired results.

[0057] The logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the logical processor 902 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. The individual components of the logical processor may optionally be distributed in two or more separate devices, which may be remotely located and / or configured for coordinated processing. Various aspects of the logical processor may be virtualized and executed by a remotely accessible networked computing device configured with a cloud computing configuration. In this case, it should be understood that these virtualization aspects are run on different physical logical processors of various different machines.

[0058] The non-volatile storage device 906 includes one or more physical devices configured to store instructions executable by a logical processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 906 can be transformed—for example, to store different data.

[0059] The non-volatile storage device 906 may include a removable and / or built-in physical device. The non-volatile storage device 906 may include an optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), a semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or a magnetic memory (e.g., a hard disk drive, a floppy disk drive, a tape drive, MRAM, etc.), or other mass storage device technology. The non-volatile storage device 906 may include a non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable, and / or content addressable device. It should be understood that the non-volatile storage device 906 is configured to save instructions even when the power to the non-volatile storage device 906 is cut off.

[0060] The volatile memory 904 may include a physical device with random access memory. The volatile memory 904 is typically used by the logical processor 902 to temporarily store information during the processing of software instructions. It should be understood that when the volatile memory 904 is powered off, the volatile memory 904 typically does not continue to store instructions.

[0061] Aspects of the logic processor 902, volatile memory 904, and non-volatile storage device 906 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0062] The terms "module," "program," and "engine" may be used to describe an aspect of a computing system 900 that is typically implemented in software by a processor to use portions of volatile memory to perform a specific function that involves converting a process that specifically configures the processor to perform that function. Thus, a module, program, or engine may be instantiated via a logical processor 902 executing instructions held by a non-volatile storage device 906 using portions of volatile memory 904. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" may include individual or groups of executable files, data files, libraries, drivers, scripts, database records, and the like.

[0063] When included, the display subsystem 908 can be used to present a visual representation of the data stored by the non-volatile storage device 906. The visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored by the non-volatile storage device, and thereby transform the state of the non-volatile storage device, the state of the display subsystem 908 can also be transformed to visually represent the changes in the underlying data. The display subsystem 908 can include one or more display devices using almost any type of technology. Such a display device can be combined with the logical processor 902, the volatile memory 904 and / or the non-volatile storage device 906 in a shared housing, or such a display device can be a peripheral display device.

[0064] When included, the input subsystem 910 may include or interface with one or more user input devices, such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the conversion and / or processing of input actions may be handled on-board or off-board. Example NUI components may include microphones for voice and / or speech recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity; and / or any other suitable sensors.

[0065] When included, the communication subsystem 912 can be configured to couple the various computing devices described herein to each other and to other devices. The communication subsystem 912 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured to communicate via a wireless telephone network, or a wired or wireless local area network or a wide area network (e.g., an HDMI connected via Wi-Fi). In some embodiments, the communication subsystem may allow the computing system 900 to send and / or receive messages to and / or from other devices via networks such as the Internet.

[0066] The following paragraphs provide additional descriptions of aspects of the present disclosure. One aspect provides a computing system for pixel-based automated testing, the automated testing being used to test a navigable simulated environment generated by a tested application. The computing system may include a computing device, the computing device including one or more processors, the one or more processors being configured to use a portion of an associated memory to execute instructions to implement an application test program during a reasoning time phase. The application test program may include a machine learning model trained to detect errors in the tested application. The application test program is configured to receive a screen image of a navigable simulated environment generated by the tested application, and to process the screen image using an object detection module to determine whether there is a key object in the screen image. If it is determined that there is a key object in the screen image, the application test program may be configured to execute an object investigation module, the object investigation module receives an image of the detected key object, and generates an investigation input for investigating the key object. If it is not determined that there is a key object in the screen image, the application test program may be configured to execute an environment exploration module, the environment exploration module receives a screen image as input, and processes the screen image to generate an environment exploration action, which will be provided to the application to be tested as a simulated user input.

[0067] In this regard, additionally or alternatively, the object detection module includes an anchor-based object detection neural network for detecting key objects in the application under test. In this regard, additionally or alternatively, the object detection neural network can be trained during a first initial training phase using a first training dataset derived from common objects in the context dataset, and the object detection neural network can be trained during a second initial training phase using a second training dataset, the second training dataset including annotated screen images of key objects in the application under test.

[0068] In this regard, additionally or alternatively, the object investigation module may include an object investigation neural network trained using a training data set including pixel-by-pixel rendered image data of the navigable simulated environment and human tester actions made in response to the image data.

[0069] In this regard, additionally or alternatively, the environment exploration module may include a target neural network and a predictor neural network, and during the exploration phase, the screen image may be input to the target neural network and the predictor neural network; and each of the target neural network and the predictor neural network maps the screen image to a corresponding embedding vector.

[0070] In this regard, additionally or alternatively, the target neural network can be randomly initialized and fixed, and the output embedding vector of the target neural network during the exploration phase is set as the true value for the predictor neural network. The predictor neural network can output a predicted embedding vector and can be trained using a distillation loss function to minimize the prediction error between the output embedding vector of the target network and the predicted embedding vector.

[0071] In this regard, additionally or alternatively, the environment exploration module may include an exploration strategy neural network that is configured to predict an environment exploration action that will produce a maximum expected reward in response to receiving a screen image as input. In this regard, additionally or alternatively, one or more processors may be configured to execute a reward function to calculate a reward for an environment exploration action taken by the exploration strategy neural network based on a prediction error for a screen image produced as a result of the environment exploration action, such that when the prediction error is larger, the reward is also larger, thereby providing a larger reward to the exploration strategy neural network when the exploration strategy neural network takes an environment exploration action that results in a screen image that is less similar to a previous screen image that has been input into the target neural network and the predictor neural network during the exploration phase.

[0072] In this regard, additionally or alternatively, the application testing program may receive only the screen image as input and be configured to execute independently of the internal state of the application under test.

[0073] On the other hand, a method for pixel-based automated testing is provided, wherein the automated testing is used to test a navigable simulated environment generated by a tested application. The method includes receiving a screen image of a navigable simulated environment generated by a tested application at one or more processors of a computing system during a reasoning time phase by an application test program, wherein the application test program includes a machine learning model trained to detect errors in the tested application. The method also includes processing the screen image using an object detection module to determine whether there is a key object in the screen image. If it is determined that there is a key object in the screen image, the method also includes executing an object investigation module, which receives an image of the detected key object and generates an investigation input for investigating the key object. If it is not determined that there is a key object in the screen image, the method also includes executing an environment exploration module, which receives a screen image as input and processes the screen image to generate an environment exploration action, which will be provided to the application to be tested as a simulated user input.

[0074] In this regard, additionally or alternatively, the method further comprises detecting key objects in the application under test via an anchor-based object detection neural network comprised by the object detection module.

[0075] In this regard, additionally or alternatively, the method further comprises training the object detection neural network during a first initial training phase using a first training dataset derived from common objects in the context dataset, and training the object detection neural network during a second initial training phase using a second training dataset, the second training dataset comprising annotated screen images of key objects in the application under test.

[0076] In this regard, additionally or alternatively, the method further comprises training an object investigation neural network comprised by the object investigation module, the object investigation neural network being trained using a training data set comprising pixel-by-pixel rendered image data of the navigable simulated environment and human tester actions made in response to the image data.

[0077] In this regard, additionally or alternatively, the environment exploration module includes a target neural network and a predictor neural network. The target neural network can be randomly initialized and fixed, and the method can also include during the exploration phase, inputting the screen image to the target neural network and the predictor neural network; and mapping the screen image to a corresponding embedding vector via each of the target neural network and the predictor neural network.

[0078] In this regard, additionally or alternatively, the method may further include, during an exploration phase, setting an output embedding vector of the target neural network as a true value for the predictor neural network, outputting a predicted embedding vector via the predictor neural network, and training the predictor neural network using a distillation loss function to minimize a prediction error between the output embedding vector of the target network and the predicted embedding vector.

[0079] In this regard, additionally or alternatively, the environment exploration module may include an exploration policy neural network configured to predict an environment exploration action that will produce a maximum expected reward in response to receiving a screen image as input.

[0080] In this regard, additionally or alternatively, the method may also include executing a reward function to calculate a reward for an environment exploration action taken by the exploration strategy neural network based on a prediction error for a screen image generated as a result of the environment exploration action, such that when the prediction error is larger, the reward is also larger, thereby providing a larger reward to the exploration strategy neural network when the exploration strategy neural network takes an environment exploration action that results in a screen image that is less similar to a previous screen image that has been input into the target neural network and the predictor neural network during the exploration phase.

[0081] In this regard, additionally or alternatively, the method may further include receiving, by the application testing program, only the screen image as input for predicting the survey input and the environment exploration action; and executing the application testing program independently of the internal state of the application under test.

[0082] On the other hand, a computing system for pixel-based automated testing is provided, and the automated testing is used to test a navigable simulated environment generated by a tested application. The computing system may include a computing device, and the computing device includes one or more processors, and the one or more processors are configured to use a portion of an associated memory to execute instructions to implement an application test program during a reasoning time phase. The application test program may include a machine learning model trained to detect errors in the tested application. The application test program is configured to receive a screen image of a navigable simulated environment generated by the tested application, and to process the screen image using an object detection module to determine whether there is a key object in the screen image. If it is not determined that there is a key object in the screen image, the application test program may be configured to execute an environment exploration module, which receives a screen image as an input, and processes the screen image to generate an environment exploration action, which will be provided to the application to be tested as a simulated user input.

[0083] In this regard, additionally or alternatively, the environment exploration module may include an exploration policy neural network configured to predict an environment exploration action that will produce a maximum expected reward in response to receiving a screen image as input, and the one or more processors may be configured to execute a reward function to calculate a reward for an environment exploration action taken by the exploration policy neural network based on a prediction error for the screen image produced as a result of the environment exploration action.

[0084] It should be understood that the configuration and / or method described herein is exemplary in nature, and because many variations are also possible, these specific embodiments or examples should not be considered restrictive. The specific routine or method described herein can represent one or more processing strategies in any number of processing strategies. Therefore, the various actions shown and / or described can be performed in the order shown and / or described, can also be performed in other orders, performed in parallel, or omitted. Similarly, the order of the above process can be changed.

[0085] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computing system for pixel-based automated testing of a navigable simulated environment generated by an application under test, comprising: A computing device comprising one or more processors configured to use portions of an associated memory to execute instructions to implement during an inference time phase: An application testing program, the application testing program comprising a machine learning model trained to detect errors in the application under test, wherein the application testing program is configured to: receiving a screen image of a navigable simulated environment generated by the application under test, and processing the screen image using an object detection module to determine whether a key object exists in the screen image; If it is determined that a key object exists in the screen image, executing an object investigation module, the object investigation module receiving the detected image of the key object and generating an investigation input for investigating the key object; as well as If it is determined that no key object exists in the screen image, an environment exploration module is executed, which receives the screen image as input and processes the screen image to generate an environment exploration action, which will be provided to the application under test as a simulated user input.

2. The computing system of claim 1, wherein: The object detection module includes an anchor-based object detection neural network for detecting key objects in the application under test.

3. The computing system of claim 2, wherein: The object detection neural network is trained during a first initial training phase using a first training dataset derived from common objects in the context dataset, and The object detection neural network is trained during a second initial training phase using a second training dataset comprising annotated screen images of key objects in the application under test.

4. The computing system of claim 1, wherein: The object investigation module includes an object investigation neural network that is trained using a training data set that includes pixel-by-pixel rendered image data of the navigable simulated environment and human tester actions made in response to the image data.

5. The computing system of claim 1, wherein: The environment exploration module includes a target neural network and a predictor neural network, During an exploration phase, the screen image is input to the target neural network and the predictor neural network; and Each of the target neural network and the predictor neural network maps the screen image to a corresponding embedding vector.

6. The computing system of claim 5, wherein: The target neural network is randomly initialized and fixed, and the output embedding vector of the target neural network during the exploration phase is set to the true value for the predictor neural network, and The predictor neural network outputs a predicted embedding vector and is trained using a distillation loss function to minimize the prediction error between the output embedding vector of the target network and the predicted embedding vector.

7. The computing system of claim 6, wherein: The environment exploration module includes an exploration policy neural network configured to predict the environment exploration action that will produce a maximum expected reward in response to receiving the screen image as input.

8. The computing system of claim 7, wherein: The one or more processors are configured to execute a reward function to calculate a reward for the environment exploration action taken by the exploration strategy neural network based on the prediction error for the screen image generated as a result of the environment exploration action, so that when the prediction error is larger, the reward is also larger, thereby providing a larger reward to the exploration strategy neural network when the exploration strategy neural network takes an environment exploration action that results in a screen image that is less similar to a previous screen image that has been input into the target neural network and the predictor neural network during the exploration phase.

9. The computing system of claim 1, wherein: The application testing program receives only the screen image as input and is configured to execute independently of the internal state of the application under test.

10. A method for pixel-based automated testing, the automated testing being used to test a navigable simulated environment generated by an application under test, the method comprising: During the inference time phase, at one or more processors of the computing system: receiving, by an application testing program, a screen image of the navigable simulated environment generated by the application under test, the application testing program including a machine learning model trained to detect errors in the application under test; Processing the screen image using an object detection module to determine whether a key object exists in the screen image; If it is determined that the key object exists in the screen image, executing an object investigation module, the object investigation module receiving the detected image of the key object and generating an investigation input for investigating the key object; as well as If it is not determined that the key object exists in the screen image, an environment exploration module is executed, which receives the screen image as input and processes the screen image to generate an environment exploration action, which will be provided to the application under test as a simulated user input.

11. The method according to claim 10, wherein: The environment exploration module includes a target neural network and a predictor neural network, wherein the target neural network is randomly initialized and fixed, and the method further includes: During an exploration phase, inputting the screen image to the target neural network and the predictor neural network; and The screen image is mapped to a corresponding embedding vector via each of the target neural network and the predictor neural network.

12. The method according to claim 11, further comprising: During the exploration phase, setting the output embedding vector of the target neural network to a true value for the predictor neural network; outputting a predicted embedding vector via the predictor neural network; as well as The predictor neural network is trained using a distillation loss function to minimize a prediction error between the output embedding vector of the target network and the predicted embedding vector.

13. The method according to claim 12, wherein: The environment exploration module includes an exploration policy neural network configured to predict the environment exploration action that will produce a maximum expected reward in response to receiving the screen image as input.

14. The method according to claim 13, further comprising: A reward function is executed to calculate a reward for the environment exploration action taken by the exploration strategy neural network based on the prediction error for the screen image generated as a result of the environment exploration action, so that when the prediction error is larger, the reward is also larger, thereby providing a larger reward to the exploration strategy neural network when the exploration strategy neural network takes an environment exploration action that results in a screen image that is less similar to a previous screen image that has been input into the target neural network and the predictor neural network during the exploration phase.

15. The method according to claim 10, further comprising: The application testing program receives only the screen image as input for predicting the survey input and the environment exploration action; as well as The application testing program is executed independently of the internal state of the application under test.

Images