Graphical user interface testing method and system and electronic equipment

Abstract

The invention discloses a graphical user interface test method and system and electronic equipment, and relates to the technical field of software test.The method comprises the steps that whether a reuse model exists in a current test environment or not is detected, and a reuse model detection result is obtained; selecting a plurality of candidate event selection strategies according to a multiplex model detection result, and determining a target event selection strategy; selecting a target control from an interactive control set of the current graphical user interface according to a target event selection strategy, and executing an event corresponding to the target control to obtain a test result of the current event; and based on the test result of the current event, updating the multiplexing model to obtain an updated multiplexing model, the updated multiplexing model being used for event selection in subsequent test rounds. According to the method and the device, the problem that test experience cannot be accumulated and cross-platform multiplexing cannot be realized is solved, so that event selection can be continuously optimized in a subsequent test round, the test efficiency and the coverage rate are remarkably improved, and the test cost is reduced.

Description

Technical Field

[0001] This application relates to the field of software testing technology, and in particular to graphical user interface testing methods, systems and electronic devices. Background Technology

[0002] As software systems become increasingly complex, graphical user interfaces (GUIs) play a crucial role in modern applications. As the direct medium for user interaction with software, the usability and stability of the GUI directly impact the overall system quality and user experience. However, compared to traditional backend logic testing, GUI testing is more susceptible to changes in controls, interface layout adjustments, and asynchronous events, leading to greater testing difficulty and higher maintenance costs.

[0003] Currently, most automated GUI testing tools only consider the coverage of single-platform testing. However, when a specific app is released on other terminals, traditional methods can only retest the newly released version, which greatly reduces testing efficiency.

[0004] Therefore, how to improve the efficiency of graphical user interface testing while ensuring test coverage is an urgent problem to be solved. Summary of the Invention

[0005] The main purpose of this application is to provide a graphical user interface testing method, system, and electronic device, aiming to solve the technical problem of how to improve the efficiency of graphical user interface testing while ensuring test coverage.

[0006] To achieve the above objectives, this application proposes a graphical user interface testing method, the method comprising: Detect whether a reused model exists in the current test environment and obtain the reused model detection results; Based on the detection results of the reuse model, multiple candidate event selection strategies are selected to determine the target event selection strategy; According to the target event selection strategy, a target control is selected from the set of interactive controls in the current graphical user interface and the event corresponding to the target control is executed to obtain the test result of the current event; Based on the test results of the current event, the reuse model is updated to obtain an updated reuse model, which is used for event selection in subsequent test rounds.

[0007] In one embodiment, the step of selecting from multiple candidate event selection strategies based on the detection results of the reuse model to determine the target event selection strategy includes: If the reuse model does not exist in the current test environment, the cold start strategy will be determined as the target event selection strategy. If a local reuse model exists in the current test environment but no external reuse model exists, then the single-machine reuse strategy will be determined as the target event selection strategy. If the current test environment has a local reuse model and at least one external reuse model, then the cross-platform reuse strategy will be determined as the target event selection strategy.

[0008] In one embodiment, the step of selecting a target control from the set of interactive controls in the current graphical user interface and executing the event corresponding to the target control according to the target event selection strategy, and obtaining the test result of the current event, includes: If the target event selection strategy is the cross-platform reuse strategy, based on each control in the set of interactive controls of the current graphical user interface, calculate the multi-dimensional attribute similarity between the internal controls of the current interface and each external control, wherein the internal controls of the current interface refer to the controls in the set of interactive controls of the current graphical user interface, and the external controls refer to the controls in the external reuse model; If the multidimensional attribute similarity exceeds a preset threshold, the current interface internal control and the corresponding external control are determined to be functionally similar controls, and the corresponding external control is defined as the target external control. Load historical test data corresponding to the target external control from the external reuse model; Based on the historical test data corresponding to the target external control, calculate the probability of jumping to a new activity page after executing the events corresponding to each of the current interface internal controls; Based on the probability of jumping to the new activity page, a target control is selected from the set of interactive controls in the current graphical user interface, and the event corresponding to the target control is executed to obtain the test result of the current event.

[0009] In one embodiment, before calculating the multidimensional attribute similarity between the internal controls and the external controls of the current interface based on each control in the set of interactive controls of the current graphical user interface, the method further includes: The multidimensional attribute vectors of all controls in the reuse model are pre-calculated to obtain a pre-calculated vector, which includes multiple attribute vectors. Construct multiple hyperplane normal vectors. For each pre-calculated attribute vector, calculate the dot product between the current attribute vector and each hyperplane normal vector, and generate the corresponding hash value based on the dot product result. Based on the hash value, the current attribute vector is assigned to the corresponding hash bucket, and similar attribute vectors are mapped to the same hash bucket or adjacent hash buckets to obtain the index structure.

[0010] In one embodiment, if the target event selection strategy is the cross-platform reuse strategy, the calculation of the multi-dimensional attribute similarity between the internal controls of the current interface and each external control is based on each control in the set of interactive controls of the current graphical user interface, including: Obtain the multidimensional properties of each control in the set of interactive controls in the current graphical user interface, and convert the multidimensional properties into a query vector; Based on the query vector and the index structure, the top-k candidate controls are retrieved from the external reuse model using a locality-sensitive hash index. Among the top-k candidate controls, the multidimensional attribute similarity between the internal controls of the current interface and each external control is obtained by calculating the similarity between the query vector and the attribute vector of each candidate control.

[0011] In one embodiment, the reuse model includes a probabilistic model and a reinforcement learning model. The step of selecting a target control from the set of interactive controls in the current graphical user interface and executing the event corresponding to the target control, based on the target event selection strategy, to obtain the test result of the current event, further includes: Determine whether there are candidate controls that are not saved in the reuse model in the set of interactive controls of the current graphical user interface. If there are, randomly select a control from the unsaved candidate controls as the target control. If there are no unsaved candidate controls, the expected value of the event corresponding to each control in the set of interactive controls of the current graphical user interface is calculated based on the probability model, and the control corresponding to the maximum expected value is selected as the target control. If all events corresponding to the interactive controls in the current graphical user interface have been executed, then based on the state action values ​​in the reinforcement learning model, the control corresponding to the maximum state action value is selected as the target control. Execute the event corresponding to the target control to obtain the test result of the current event.

[0012] In one embodiment, the step of calculating the expected value of the event corresponding to each control in the set of interactive controls of the current graphical user interface based on the probability model, and selecting the control corresponding to the maximum expected value as the target control, includes: For each control in the set of interactive controls in the current graphical user interface, based on the historical test data in the probability model, the probability of jumping to an unaccessed control after executing the event corresponding to the current control is calculated. Based on the probability of jumping to an unaccessed control, calculate the expected increase in activity coverage after executing the event corresponding to each control; The expected value of the increased activity coverage is normalized by using the softmax function to obtain the normalized jump probability of the event corresponding to each control. Based on the normalized jump probability, the control with the highest normalized jump probability is selected from the set of interactive controls in the current graphical user interface as the target control.

[0013] In one embodiment, updating the reuse model based on the test result of the current event to obtain an updated reuse model, the updated reuse model being used for event selection in subsequent test rounds, includes: Based on the test results of the current event, extract the multi-dimensional attributes of the corresponding control. The multi-dimensional attributes include text content, resource ID, activity name, and icon code. Based on the multidimensional attributes of the corresponding control, the correspondence between events and activities after navigation and controls, and the access count, the reuse model is updated, and the access count is also updated. Based on the event execution results, the Q-value of the event is updated using a reinforcement learning algorithm.

[0014] Furthermore, to achieve the above objectives, this application also proposes a graphical user interface (GUI) testing system, which includes: The model detection module is used to detect whether there are reused models in the current test environment and obtain the reused model detection results; The strategy determination module is used to select from multiple candidate event selection strategies based on the detection results of the reuse model, and determine the target event selection strategy. The event testing module is used to select a target control from the set of interactive controls in the current graphical user interface according to the target event selection strategy, execute the event corresponding to the target control, and obtain the test result of the current event. The model update module is used to update the reuse model based on the test results of the current event, so as to obtain the updated reuse model, which is used for event selection in subsequent test rounds.

[0015] In addition, to achieve the above objectives, this application also proposes an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the graphical user interface testing method described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: The system detects whether a reused model exists in the current test environment and obtains the reused model detection results. By detecting the existence of a reused model in the current test environment, it solves the problem that traditional GUI testing tools cannot utilize historical testing experience and must start from scratch for each test, thus achieving an initial improvement in testing efficiency.

[0017] Based on the reusable model's detection results, multiple candidate event selection strategies are chosen to determine the target event selection strategy. By employing a technique that adaptively selects the target event selection strategy based on the reusable model's detection results, the problem of a single testing strategy and its inability to flexibly adapt to different testing scenarios is solved. This allows the testing process to dynamically adjust the strategy based on the presence or absence of the model, enhancing the adaptability and relevance of the testing.

[0018] Based on the target event selection strategy, a target control is selected from the set of interactive controls in the current graphical user interface, and the corresponding event of the target control is executed to obtain the test result of the current event. By employing a technique that selects and executes the target event from the set of interactive controls based on the target event selection strategy, the problem of repeatedly triggering the same control during event selection is solved, thus improving test coverage.

[0019] Based on the test results of the current event, the reuse model is updated to obtain the updated reuse model, which is used for event selection in subsequent test rounds. Updating the reuse model based on test results solves the problems of insufficient test experience accumulation and cross-platform reuse. This allows subsequent test rounds to utilize the updated model to optimize event selection, achieving continuous self-improvement of the testing process, significantly improving test efficiency and coverage, while also supporting cross-platform test experience transfer and reducing testing costs. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the first embodiment of the graphical user interface testing method of this application; Figure 2 This is a flowchart illustrating the complete test rounds in the embodiments of this application; Figure 3This is a schematic diagram illustrating the four-round cumulative cross-end reuse scenario in an embodiment of this application; Figure 4 This is a schematic diagram of the module structure of the graphical user interface test system of this application.

[0023] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0024] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0025] This application provides a graphical user interface (GUI) testing method, which can be applied to a GUI testing system. (Refer to...) Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the graphical user interface testing method of this application.

[0026] In this embodiment, the graphical user interface testing method includes steps S10 to S40: Step S10: Detect whether a reused model exists in the current test environment and obtain the reused model detection result.

[0027] It should be noted that the reused model can be either a Probability Model (PM model) or a reinforcement learning model, both trained based on historical test data. The PM model stores historical test data, including statistical information from events to controls, such as access counts and jump probabilities. The PM model supports cross-platform reuse by recording the multimodal attributes of controls to transfer testing experience across different terminals. The reinforcement learning model can reuse decision-making experience accumulated through reinforcement learning. The reused model detection result refers to the result obtained at the start of the test by detecting whether an external reused model exists in the current test environment.

[0028] For example, when building a probabilistic model, instead of recording statistics for "action → activity," the statistical granularity is refined to "action → control," giving each control an independent access count. To support cross-platform reuse, an ActionInfo can be defined for each action, and a WidgetInfo can be defined for each control, containing attributes such as text, resource ID, associated activity name, and icon code. Furthermore, at the start of each test, the system automatically detects and loads any external reused models present on the current test device.

[0029] Step S20: Based on the detection results of the reuse model, select multiple candidate event selection strategies to determine the target event selection strategy.

[0030] It should be noted that the candidate event selection strategy can include multiple priority strategies provided in the event decision algorithm, as well as event selection strategies corresponding to different modes. Specifically, it can include the cold start strategy in the cold start mode, the single-machine reuse strategy in the single-machine reuse mode, and the cross-platform reuse strategy in the cross-platform reuse mode.

[0031] Step S30: Based on the target event selection strategy, select the target control from the set of interactive controls in the current graphical user interface and execute the event corresponding to the target control to obtain the test result of the current event.

[0032] It should be noted that a Graphical User Interface (GUI) refers to the visual interface through which users interact in a software application, including interactive controls such as buttons and input boxes. The set of interactive controls refers to the collection of all user-operable controls in the current GUI interface. A target control refers to a specific control selected from the set of interactive controls based on a target event selection strategy; its corresponding event will be executed for testing. The interactive action corresponding to the target control in the current event, such as clicking or swiping, will trigger a change in the interface state when the event is executed. It can be understood that events are bound to controls, and testing explores new interfaces by executing events. This embodiment of the application simulates user operations to verify interface functionality by selecting a target control from the set of interactive controls in the current graphical user interface and executing the event corresponding to the target control.

[0033] Step S40: Based on the test results of the current event, update the reuse model to obtain the updated reuse model. The updated reuse model is used for event selection in subsequent test rounds.

[0034] It should be noted that the test result of the current event refers to the result obtained after executing the event corresponding to the target control, which may include the activity page to be jumped to, newly exposed control information, etc.

[0035] In this embodiment, by detecting the existence of a reusable model in the current test environment, the problem of traditional GUI testing tools being unable to utilize historical testing experience and having to start from scratch for each test is solved, thus achieving an initial improvement in testing efficiency. By employing a technique that adaptively selects target event selection strategies based on the reusable model detection results, the problem of single testing strategies and inflexibility in handling different testing scenarios is solved. This allows the testing process to dynamically adjust strategies based on the presence or absence of a model, enhancing the adaptability and relevance of the test. Furthermore, by employing a technique that selects and executes target events from the set of interactive controls based on the target event selection strategy, the problem of event selection easily triggering the same controls repeatedly is solved, improving test coverage. Updating the reusable model based on test results solves the problem of the inability to accumulate testing experience and reuse it across platforms. This allows subsequent test rounds to utilize the updated model to optimize event selection, achieving continuous self-improvement of the testing process and significantly improving testing efficiency and coverage. Simultaneously, it supports cross-platform testing experience migration, reducing testing costs.

[0036] In one implementation, step S20 includes: if there is no reuse model in the current test environment, then the cold start strategy is determined as the target event selection strategy; if there is a local reuse model and no external reuse model in the current test environment, then the single-machine reuse strategy is determined as the target event selection strategy; if there is a local reuse model and at least one external reuse model in the current test environment, then the cross-platform reuse strategy is determined as the target event selection strategy.

[0037] It should be noted that local reuse models refer to the probabilistic models and reinforcement learning models stored on the current test device, including historical test data accumulated when the device previously tested the same application. External reuse models refer to probabilistic models and reinforcement learning models from other terminal devices, storing historical test data of the same application on other platforms. The cold start strategy is adopted when no reuse models are detected. The cold start strategy can prioritize randomly selecting events in the current interface that are not recorded in the probabilistic models to maximize the possibility of exploring new interface states. The single-machine reuse strategy is adopted when only local reuse models exist. It can load historical data from the local model through hash value matching, calculate the expected value of the event jump, and select the event with the highest expected value for execution. The cross-platform reuse strategy is adopted when local reuse models and at least one external reuse model exist. It loads cross-platform historical data after a successful match by calculating the multimodal attribute similarity between the current event and the event in the external model.

[0038] In this embodiment, the technique of adaptively selecting testing strategies based on the type of reused models solves the problem of traditional GUI testing tools having a single strategy and being unable to distinguish between local and cross-platform reuse scenarios, thus achieving precise matching between test modes and model resources. By employing a cold-start strategy when no reused model exists, the problem of having no historical data for reference in the initial testing phase is solved. A basic probability model is quickly built by randomly exploring unrecorded events, laying a data foundation for subsequent reuse. By employing a single-machine reuse strategy when only a local model exists, the problem of low utilization of single-platform testing experience is solved, significantly improving single-platform testing efficiency. By employing a cross-platform reuse strategy when a cross-platform model exists, the problem of the inability to transfer testing experience between different terminals is solved. By directly reusing external testing history, redundant exploration is avoided, significantly reducing cross-platform testing costs. Through strategy layering and scenario adaptation mechanisms, the testing tool can optimize resource utilization in cold-start, single-machine reuse, and cross-platform reuse scenarios, improving overall test coverage and convergence speed.

[0039] Based on the above embodiments and implementation methods, in the second embodiment, step S30 includes: Step A301: If the target event selection strategy is a cross-platform reuse strategy, calculate the multi-dimensional attribute similarity between the internal controls of the current interface and each external control based on each control in the set of interactive controls of the current graphical user interface.

[0040] It should be noted that the internal controls of the current interface refer to the controls in the set of interactive controls of the current graphical user interface, while the external controls refer to the controls in the external reuse model. For example, taking control 'a' in the set of interactive controls of the current graphical user interface as the internal control of the current interface, calculating the multi-dimensional attribute similarity between the internal control of the current interface and each external control yields multiple multi-dimensional attribute similarity values. These multi-dimensional attribute similarity values ​​are taken as a subset of 'a', denoted as A. Similarly, other controls b and c will also obtain corresponding multi-dimensional attribute similarity subsets B and C.

[0041] For example, multidimensional attributes may include activity name, text, resource-id, action type, icon code, etc.

[0042] Step A302: If the multidimensional attribute similarity exceeds a preset threshold, then the internal control of the current interface and the corresponding external control are determined to be functionally similar controls, and the corresponding external control is defined as the target external control.

[0043] It's worth noting that multidimensional attributes support cross-platform alignment. For example, the same function button may have different resource IDs on Android and iOS platforms, but may have similar text or icons. By calculating a comprehensive similarity score based on multidimensional attributes, controls are considered functionally equivalent when the similarity exceeds a set threshold, thus allowing the reuse of their historical access probabilities and reward values.

[0044] For example, if, among subsets A, B, and C, A has a multidimensional attribute similarity exceeding a preset threshold, then it is determined that the internal control 'a' of the current interface is similar to the corresponding external control. For controls with similar functions, and external controls Defined as the target external control.

[0045] For example, upon reaching an activity page, the multi-dimensional attributes of all controls on the current page are automatically extracted to support similarity calculation. This can be achieved by loading the text, resource-id, activity name, and icon data corresponding to each control stored in the reuse model, and then calculating the weighted similarity of the above four attributes between the event on the current page and the event in the reuse model. The calculation formula is as follows:

[0046] in, These represent the weights of the four parameters. It is a control Text attribute similarity, It is a control Resource-id attribute similarity It is a control Activity Name attribute similarity It is a control The similarity of the Icon attribute is used to determine if the similarity is greater than a preset threshold. If so, the two events are considered to be functionally similar, and the match is successful.

[0047] Step A303: Load historical test data corresponding to the target external control from the external reuse model.

[0048] It should be noted that the target external control refers to an external control in the external reuse model that has similar functionality to the internal controls of the current interface and is selected as the object for data reuse. Historical test data refers to the test history information associated with the target external control in the external reuse model, including the jump probability, access count, and reinforcement learning Q-value after the control is triggered.

[0049] Step A304: Based on the historical test data corresponding to the target external control, calculate the probability of jumping to the new activity page after executing the corresponding event of each current interface internal control.

[0050] It should be noted that the probability of being redirected to a new activity page refers to the likelihood that, after executing the event corresponding to a certain control, navigation will lead to an activity page that has not been visited before.

[0051] For example, test data for the corresponding event is loaded from the reuse model, and the probability of being redirected to a new activity page after the event is executed is successfully calculated using this test data. The calculation formula is as follows:

[0052] =

[0053] =

[0054] in, Indicates execution complete The probability of increased activity coverage afterward. Indicates execution complete The expected value of subsequent activity coverage increase, Indicates execution complete Then the controls can be exposed. The probability of this can be achieved using the method selectUnperformedActionInReuseModel.

[0055] Step A305: Based on the probability of jumping to a new activity page, select the target control from the set of interactive controls in the current graphical user interface, execute the event corresponding to the target control, and obtain the test result of the current event.

[0056] It should be noted that the target control refers to the final control selected from the current set of interactive controls, whose corresponding event is executed. The selection criterion is to maximize the jump probability. The test result of the current event can be understood as the output obtained after executing the target control event, including whether the jump to the new activity was successful, the newly exposed set of controls, etc., which is used to evaluate test coverage and update the model.

[0057] In this embodiment, by calculating the multi-dimensional attribute similarity between internal and external controls of the current interface under a cross-platform reuse strategy, the problem of inconsistent control attributes and inability to directly match due to platform differences in traditional GUI testing tools is solved, thus achieving initial alignment of cross-platform testing experience. By using a preset threshold to determine functionally similar controls, the problem of ambiguous judgment of cross-platform control functional equivalence is solved, avoiding exploration bias caused by mismatches. By employing the technique of loading historical test data corresponding to the target external control from an external reuse model, the problem of historical information not being directly usable in cross-platform testing is solved, allowing current testing to draw on the exploration experience of external terminals and reducing repetitive operations. Based on historical data, the probability of jumping to a new activity page is calculated, and probabilistic guidance prioritizes high-reward events, optimizing the test path and solving the problem of slow coverage improvement under complex interfaces, significantly accelerating the exploration efficiency of new activities. Through multi-dimensional attribute similarity calculation and cross-platform data reuse mechanism, seamless transfer of testing experience is achieved, reducing multi-platform testing costs while improving the convergence speed of coverage.

[0058] In one implementation, before step A301, the method further includes: pre-computing the multidimensional attribute vectors of all controls in the reuse model to obtain pre-computed vectors, the pre-computed vectors including multiple attribute vectors; constructing multiple hyperplane normal vectors, for each pre-computed attribute vector, calculating the dot product between the current attribute vector and each hyperplane normal vector, and generating a corresponding hash value based on the dot product result; allocating the current attribute vector to the corresponding hash bucket based on the hash value, and mapping similar attribute vectors to the same hash bucket or adjacent hash buckets to obtain an index structure.

[0059] It should be noted that multidimensional attribute vectors refer to the numerical vector representations of multiple attributes of a control. Each attribute can be converted into a high-dimensional embedding vector using models such as BERT or CLIP for similarity calculation. Pre-computed vectors refer to the set of results from batch computation of multidimensional attribute vectors for all controls in the reused model before testing begins. Pre-computation avoids the overhead of real-time vector generation and is directly stored in a file for loading during testing. An attribute vector is the vector representation of a single control attribute; for example, a text attribute corresponds to one vector, and a resource ID attribute corresponds to another. Multidimensional attribute vectors are the collection of these attribute vectors. Hyperplane normal vectors refer to the randomly generated normal vectors in the LSH algorithm, used to define the hash function. Each hyperplane normal vector is sampled from a standard normal distribution, and its dot product is combined with the input vector to generate a hash value. The index structure refers to a fast retrieval system built on LSH, containing multiple hash tables and corresponding hash buckets. This structure allows direct location of the candidate vector set through hash values, avoiding global scanning. The query vector refers to the multidimensional attribute vector of the controls in the current graphical user interface. It is generated in real time during testing or loaded from a pre-computed vector and is used to retrieve similar external controls in the LSH index.

[0060] For example, before testing an app, all action and widget vectors stored in the external and local models corresponding to the current app can be calculated and saved to files, and automatically loaded into memory before each test, thereby reducing the performance overhead of real-time computation. Search efficiency can be improved by introducing Locality Sensitive Hashing (LSH). Specifically, similar objects can be placed in a hash bucket, and the number of elements in the bucket can be controlled by setting the bucket size. The dot product formula is defined as follows:

[0061]

[0062] The formula for calculating hash values ​​is defined as follows:

[0063] Where v represents the input vector to be hashed, i.e., the query vector, with dimension d. Let i be the i-th component of vector v, where i∈{0,…,d-1}. Let be the normal vector of the b-th hyperplane in the l-th hash table, with length d, which can be independently sampled from the standard normal distribution N(0,1). Hyperplane vector The i-th component.

[0064] Based on the above implementation method, step A301 includes: obtaining the multidimensional attributes of each control in the set of interactive controls of the current graphical user interface, and converting the multidimensional attributes into query vectors; according to the query vectors, retrieving the top-k candidate controls from the external reuse model through a local sensitive hash index based on the index structure; and among the top-k candidate controls, obtaining the multidimensional attribute similarity between the internal controls of the current interface and each external control by calculating the similarity between the query vector and the attribute vector of each candidate control.

[0065] For example, for each control in the set of interactive controls in the current graphical user interface, the top-k most similar objects in the external reuse model can be quickly found through its LSH index, and then the most similar match can be found among these top-k objects.

[0066] In this implementation, by pre-computing the multidimensional attribute vectors of all controls in the reused model, the high performance overhead of real-time vector calculation when calling the embedded model is solved, thus enabling offline computation and reducing latency during testing. The technique of constructing hyperplane normal vectors and generating hash values ​​based on dot products solves the problem of high computational complexity for high-dimensional vector similarity. Binary hashing discretizes continuous similarity, laying the foundation for fast indexing. The technique of allocating attribute vectors to hash buckets based on hash values ​​and constructing an index structure solves the problem of low efficiency in comparing a large number of controls individually in the external model. Bucket storage narrows the search scope from global to local, significantly improving matching speed. Furthermore, the retrieval of top-k candidate controls based on query vectors and LSH indexes solves the problem of high real-time requirements for similarity calculation in cross-platform scenarios. Approximate search quickly filters the candidate set, avoiding redundant computation. The technique of calculating precise similarity among the top-k candidate controls solves the precision loss problem that LSH may cause, ensuring accuracy in functional similarity determination through refined comparison while maintaining efficiency.

[0067] In the third embodiment, step S30 further includes: Step B301: Determine whether there are candidate controls in the current set of interactive controls in the graphical user interface that are not saved in the reuse model. If so, randomly select a control from the unsaved candidate controls as the target control.

[0068] It should be noted that candidate controls not saved in the reuse model refer to controls in the current set of interactive controls in the graphical user interface that are not recorded in the PM model.

[0069] For example, suppose that at step t, the test system is in the interface state. This includes the identifier of the current activity page. The current collection of interactive controls In each round of testing, the current interface state is first obtained. Extract the set of interactive controls. Then, based on the event decision algorithm, an event is selected to be executed on the current screen, and the set of activities that have been accessed is recorded. and control collection The system first checks if there are any events on the current page that are not saved in the local probability model or if no similar events are found in the external model. If so, the system will prioritize these events that have not yet been executed, as they are more likely to reach previously unvisited activity pages. The specific selection method involves first adding the events that meet the above requirements to a candidate set, and then randomly selecting one. This part is implemented by the method `selectUnperformedActionNotInReuseModel`.

[0070] Step B302: If there are no unsaved candidate controls, calculate the expected value of the event corresponding to each control in the set of interactive controls in the current graphical user interface based on the probability model, and select the control with the largest expected value as the target control.

[0071] It should be noted that, for each control in the current set of interactive controls in the graphical user interface, based on historical test data in the probability model, the probability of jumping to an unaccessed control after executing the event corresponding to the current control is calculated; based on the probability of jumping to an unaccessed control, the expected value of the increase in activity coverage after executing the event corresponding to each control is calculated; the expected value of the increase in activity coverage is normalized by the softmax function to obtain the normalized jump probability of the event corresponding to each control; based on the normalized jump probability, the control with the highest normalized jump probability is selected from the current set of interactive controls in the graphical user interface as the target control.

[0072] It's important to note that the probability of jumping to an unvisited control refers to the likelihood that executing an event corresponding to a control will expose controls that haven't appeared in the current test round. The normalized jump probability, processed by the softmax function, represents the relative likelihood of each event jumping to a new activity, used to select the optimal event from candidate events. The new probability model can be understood as selecting current candidate controls based on previously explored pages, choosing the controls most likely to trigger on unexplored pages, thus improving Activity coverage in the test results. When some components on a page are not triggered in this test round, this embodiment calls the probability model to select the best control from historical experience. For example, for a potentially interactive event e, the controls that event e could previously trigger can be read based on previous exploration, and the controls explored in this round are compared with historical data. If the current test round does not trigger any controls previously triggered by event e, and compared to other events on the current page, event e still has the largest number of untriggered controls, then event e will be selected by the probability model. The probability model calculates the probability of each e, where E(e) represents the proportion of controls that were triggered before event e but were not triggered in this round.

[0073] Step B303: If all events corresponding to the interactive controls in the current graphical user interface have been executed, then based on the state action values ​​in the reinforcement learning model, select the control corresponding to the maximum state action value as the target control.

[0074] It should be noted that the reinforcement learning model refers to the learning component integrated in the embodiments of this application, which records the long-term reward of an event in a specific state through the Q-value, and is used to guide decision-making when all events have been explored.

[0075] For example, after each event is executed, the current probability model and the Q-value corresponding to that event are updated synchronously. If all events corresponding to interactive controls in the current page's set have been explored, then... - Greedy algorithms or Gumbel-Softmax strategies sample the optimal action from the Q-value.

[0076] Step B304: Execute the event corresponding to the target control and obtain the test result of the current event.

[0077] It should be noted that the test result of the current event refers to the output obtained after executing the event corresponding to the target control. It may include whether to jump to a new activity, a new set of exposed controls, and changes in the interface state, which are used to update the probability model and reinforcement learning model.

[0078] In this embodiment, by determining whether there are candidate controls not saved in the reuse model in the current set of interactive controls and randomly selecting one, the problem of lacking historical data guidance in the initial testing phase or when exploring new interfaces is solved. By employing a technique that calculates the expected value of the event corresponding to the control based on a probability model and selects the maximum value, the problem of insufficient utilization of historical testing experience is addressed. Data-driven, precise decision-making is achieved by statistically analyzing the probability of jumping to an unvisited control and the expected value of activity coverage. The use of the softmax function to normalize the expected value solves the problem of inconsistent scales when comparing probabilities of multiple events, ensuring a reasonable and interpretable probability distribution and improving selection stability. When all events have been explored, the target control is selected based on the state and action values ​​of the reinforcement learning model, solving the efficiency degradation caused by repeated exploration. Through multi-strategy layered fusion, dynamic optimization of the test path is achieved, significantly improving activity coverage and testing efficiency while reducing the risk of insufficient local exploration.

[0079] To make the embodiments of this application clearer, as follows: Figure 2 In one embodiment, the fusion selection of steps A301 to A305 and steps B301 to B304 is further explained. For example, assume that in step t, the test system is in the interface state. This includes: the current activity / page identifier. The current collection of interactive controls In each round of testing, the current interface state is first obtained. Extract the set of interactive controls. The event decision algorithm selects one event to execute on the current screen and records the set of activities that have been accessed. and control collection The event decision algorithm can be referenced in steps B301 to B304. For example, GUI testing is divided into the following modes: cold start mode, single-machine reuse mode, and cross-platform reuse mode. In single-machine reuse mode, there is no external model data, and test data reuse is performed on a single platform. In cross-platform reuse mode, there may be test data from one or more other terminal devices, and test data reuse is performed across platforms. After the test starts, all reused models in the current device are first checked. If no reused model exists in the current device, our testing tool will enter cold start mode, corresponding to the cold start strategy, i.e., step B301, which uses the highest priority strategy in the event decision algorithm, that is, randomly selecting an unexecuted event to execute. It should be noted that during cold start, the events on each page are unexecuted, and selecting unexecuted events is more likely to cover new activities. In cold start mode, since the events newly saved to the reused model during the current test have been executed at least once, step B302 will not be triggered.

[0080] If the current device only has a reused model for the local platform, then the test will enter a single-machine reuse mode. In single-machine reuse mode, the focus is on reusing historical test data stored in the probability model to quickly increase test coverage. Events are selected using the strategy in step B302 of the event decision algorithm. Based on the hash value of each event on the current page, the test history information is searched in the probability model. If a hash value match is found, the test data is loaded, and the probability of being redirected to a new activity page after executing the event is successfully calculated using this test data. After calculating the expected value of all events that meet the conditions on the current page, the event with the highest expected value is selected for execution. By utilizing test history information, this round of testing can quickly cover activities not explored in previous tests, improving testing efficiency and test coverage.

[0081] If the current device not only has the reuse model of the local platform, but also has the reuse model of one or more other terminals, it will enter the cross-terminal reuse mode, that is, execute steps A301 to A305.

[0082] In one implementation, step S40 includes: extracting multidimensional attributes of the corresponding control based on the test results of the current event, the multidimensional attributes including text content, resource ID, activity name, and icon code; updating the reuse model and updating the access count based on the multidimensional attributes of the corresponding control, the correspondence between the event and the activity and control after the jump, and the access count; and updating the Q value of the event through a reinforcement learning algorithm based on the event execution results.

[0083] It's important to note that the text content (text) refers to the text information displayed on the control, such as button labels or input box tooltips, used to determine control functionality through semantic similarity. The resource ID (resource-id) refers to the unique resource identifier of the control in the application code, used to precisely match instances of the same control in different contexts. The activity name (activity name) refers to the name of the graphical user interface activity or page to which the control belongs, used to identify the control's context. The icon encoding (icon) refers to the encoding form of the control's icon image, such as vector embedding, which can support icon similarity calculations through visual features. The correspondence between events and the activities and controls after the jump refers to the mapping of interface state changes triggered after an event is executed, including the target activity to which the event jumps, and the set of controls that newly appear in that activity. This relationship is used to update jump statistics in the probability model. The access count refers to the statistics of the number of times a control or event is successfully accessed or triggered, used to calculate jump probabilities and evaluate the importance of controls. The Q-value refers to the state-action value in the reinforcement learning model, representing the expected cumulative reward that can be obtained by executing a certain event in a specific interface state, used to guide event selection decisions in subsequent tests.

[0084] For example, after each round of testing, the jump information and Q-value of the event are automatically updated using the Sarsa N-Step algorithm and saved in the reuse model of the current platform. N-step Sarsa is a reinforcement learning algorithm that combines temporal difference and Monte Carlo methods, estimating the current state-action value by observing the reward and state-action pairs over n steps. Before each save, the text, resource-id, activity name, and icon data of the target control corresponding to the event are automatically extracted, as well as the text, resource-id, activity name, icon data, and jump count of all controls on the activity page to which the event is jumped, preparing for cross-platform matching. The reinforcement learning agent calculates the reward based on the current exploration page and updates the Q-value of the state N steps ago, storing the Q-value in a temporary Q-table. The Q-values ​​of the states recorded in the Q-table are the basis for the reinforcement learning model to make the next decision. The state represents the component triggered at a certain time step and the page where the component is located. The reinforcement learning model uses the Q-table to rank the components to be triggered next, selecting the component most likely to trigger a new page. The formula for calculating the q value at time step t is expressed as:

[0085] +

[0086] in, Representative event Q value, It's the learning rate. This represents the cumulative reward over n steps.

[0087] In this implementation, multi-dimensional attributes of corresponding controls are extracted based on the test results of the current event. This solves the problem of incomplete or missing key attribute information extraction during the testing process, thereby ensuring the comprehensiveness and accuracy of the data required for model updates and laying the foundation for cross-platform matching. By using multi-dimensional attributes, the correspondence between events and subsequent activities and controls, and the access count update reuse model, the problem of the inability to dynamically accumulate and refine historical testing experience in traditional GUI testing tools is solved. Real-time recording of control-level jump statistics and attribute information enables continuous optimization of testing experience and fine-grained updates to the reuse model. By updating the Q-value of events through reinforcement learning algorithms, the problem of event selection strategies not being able to adaptively adjust based on test feedback is solved. Dynamic optimization of the Q-value using a reward mechanism enables the testing tool to more intelligently balance exploration and utilization. Through the synergistic effect of multi-dimensional attribute extraction, dynamic model updates, and reinforcement learning, the complete preservation and intelligent evolution of testing experience are achieved, significantly improving the convergence speed of test coverage and the adaptability of cross-platform testing.

[0088] Based on the above embodiments and implementation methods, this application conducts experimental verification. After multiple rounds of cumulative testing, the cumulative number of cross-platform reuse exploration activities continues to increase. For example, details of the four rounds of cumulative cross-platform reuse are as follows: Figure 3 As shown.

[0089] Based on the same inventive concept, such as Figure 4 As shown, this application also provides a graphical user interface (GUI) testing system, which includes: Model detection module 10 is used to detect whether there are reused models in the current test environment and obtain the reused model detection results; The strategy determination module 20 is used to select from multiple candidate event selection strategies based on the detection results of the reuse model, and determine the target event selection strategy. The event testing module 30 is used to select a target control from the set of interactive controls in the current graphical user interface according to the target event selection strategy and execute the event corresponding to the target control to obtain the test result of the current event. The model update module 40 is used to update the reuse model based on the test results of the current event to obtain the updated reuse model, which is used for event selection in subsequent test rounds.

[0090] It should be noted that the graphical user interface testing system provided in this application embodiment and the graphical user interface testing method provided in this application embodiment are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned graphical user interface testing method, and the repeated parts will not be described again.

[0091] In some embodiments, an electronic device provided in this application includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the above-described graphical user interface testing method.

[0092] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0093] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0094] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A graphical user interface testing method, characterized in that, The method includes: Detect whether a reused model exists in the current test environment and obtain the reused model detection results; Based on the detection results of the reuse model, multiple candidate event selection strategies are selected to determine the target event selection strategy; According to the target event selection strategy, a target control is selected from the set of interactive controls in the current graphical user interface and the event corresponding to the target control is executed to obtain the test result of the current event; Based on the test results of the current event, the reuse model is updated to obtain an updated reuse model, which is used for event selection in subsequent test rounds.

2. The method as described in claim 1, characterized in that, The step of selecting from multiple candidate event selection strategies based on the detection results of the reuse model, and determining the target event selection strategy, includes: If the reuse model does not exist in the current test environment, the cold start strategy will be determined as the target event selection strategy. If a local reuse model exists in the current test environment but no external reuse model exists, then the single-machine reuse strategy will be determined as the target event selection strategy. If a local reuse model and at least one external reuse model exist in the current test environment, then the cross-platform reuse strategy will be determined as the target event selection strategy.

3. The method as described in claim 2, characterized in that, The step of selecting a target control from the set of interactive controls in the current graphical user interface according to the target event selection strategy and executing the event corresponding to the target control to obtain the test result of the current event includes: If the target event selection strategy is the cross-platform reuse strategy, based on each control in the set of interactive controls of the current graphical user interface, calculate the multi-dimensional attribute similarity between the internal controls of the current interface and each external control, wherein the internal controls of the current interface refer to the controls in the set of interactive controls of the current graphical user interface, and the external controls refer to the controls in the external reuse model; If the multidimensional attribute similarity exceeds a preset threshold, the current interface internal control and the corresponding external control are determined to be functionally similar controls, and the corresponding external control is defined as the target external control. Load historical test data corresponding to the target external control from the external reuse model; Based on the historical test data corresponding to the target external control, calculate the probability of jumping to a new activity page after executing the events corresponding to each of the current interface internal controls; Based on the probability of jumping to the new activity page, a target control is selected from the set of interactive controls in the current graphical user interface, and the event corresponding to the target control is executed to obtain the test result of the current event.

4. The method as described in claim 3, characterized in that, Before calculating the multidimensional attribute similarity between the internal controls and each external control in the set of interactive controls based on the current graphical user interface, the method further includes: The multidimensional attribute vectors of all controls in the reuse model are pre-calculated to obtain a pre-calculated vector, which includes multiple attribute vectors. Construct multiple hyperplane normal vectors. For each pre-calculated attribute vector, calculate the dot product between the current attribute vector and each hyperplane normal vector, and generate the corresponding hash value based on the dot product result. Based on the hash value, the current attribute vector is assigned to the corresponding hash bucket, and similar attribute vectors are mapped to the same hash bucket or adjacent hash buckets to obtain the index structure.

5. The method as described in claim 4, characterized in that, If the target event selection strategy is the cross-platform reuse strategy, based on each control in the set of interactive controls in the current graphical user interface, the multi-dimensional attribute similarity between the internal controls of the current interface and each external control is calculated, including: Obtain the multidimensional properties of each control in the set of interactive controls in the current graphical user interface, and convert the multidimensional properties into a query vector; Based on the query vector and the index structure, the top-k candidate controls are retrieved from the external reuse model using a locality-sensitive hash index. Among the top-k candidate controls, the multidimensional attribute similarity between the internal controls of the current interface and each external control is obtained by calculating the similarity between the query vector and the attribute vector of each candidate control.

6. The method as described in claim 1, characterized in that, The reuse model includes a probabilistic model and a reinforcement learning model. The step of selecting a target control from the current set of interactive controls in the graphical user interface and executing the event corresponding to the target control, based on the target event selection strategy, to obtain the test result of the current event, further includes: Determine whether there are candidate controls that are not saved in the reuse model in the set of interactive controls of the current graphical user interface. If there are, randomly select a control from the unsaved candidate controls as the target control. If there are no unsaved candidate controls, the expected value of the event corresponding to each control in the set of interactive controls of the current graphical user interface is calculated based on the probability model, and the control corresponding to the maximum expected value is selected as the target control. If all events corresponding to all controls in the current set of interactive controls in the graphical user interface have been executed, then based on the state action values ​​in the reinforcement learning model, the control corresponding to the maximum state action value is selected as the target control. Execute the event corresponding to the target control to obtain the test result of the current event.

7. The method as described in claim 6, characterized in that, The step of calculating the expected value of the event corresponding to each control in the set of interactive controls of the current graphical user interface based on the probability model, and selecting the control corresponding to the maximum expected value as the target control, includes: For each control in the set of interactive controls in the current graphical user interface, based on the historical test data in the probability model, the probability of jumping to an unaccessed control after executing the event corresponding to the current control is calculated. Based on the probability of jumping to an unaccessed control, calculate the expected increase in activity coverage after executing the event corresponding to each control; The expected value of the increased activity coverage is normalized by using the softmax function to obtain the normalized jump probability of the event corresponding to each control. Based on the normalized jump probability, the control with the highest normalized jump probability is selected from the set of interactive controls in the current graphical user interface as the target control.

8. The method as described in claim 1, characterized in that, Based on the test results of the current event, the reuse model is updated to obtain an updated reuse model. The updated reuse model is used for event selection in subsequent test rounds, including: Based on the test results of the current event, extract the multi-dimensional attributes of the corresponding control. The multi-dimensional attributes include text content, resource ID, activity name, and icon code. Based on the multidimensional attributes of the corresponding control, the correspondence between events and activities after navigation and controls, and the access count, the reuse model is updated, and the access count is also updated. Based on the event execution results, the Q-value of the event is updated using a reinforcement learning algorithm.

9. A graphical user interface testing system, characterized in that, The graphical user interface testing system includes: The model detection module is used to detect whether there are reused models in the current test environment and obtain the reused model detection results; The strategy determination module is used to select from multiple candidate event selection strategies based on the detection results of the reuse model, and determine the target event selection strategy. The event testing module is used to select a target control from the set of interactive controls in the current graphical user interface according to the target event selection strategy, execute the event corresponding to the target control, and obtain the test result of the current event. The model update module is used to update the reuse model based on the test results of the current event, so as to obtain the updated reuse model, which is used for event selection in subsequent test rounds.

10. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the graphical user interface testing method as described in any one of claims 1 to 8.

Images