A method and apparatus for page testing
By performing page segmentation graph recognition and multi-agent collaborative operation, interactive elements are dynamically identified, solving the problems of low efficiency and poor adaptability in existing testing methods, and achieving efficient and accurate application testing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-11
- Publication Date
- 2026-06-23
AI Technical Summary
Existing mobile application testing methods rely on manual operation, which is inefficient and generally ineffective, and cannot adapt to changing application scenarios and personalized needs.
By dividing the page image under test into multiple segmentation maps, interactive elements are identified, important elements are dynamically identified and filtered, exploration paths are generated, and automated testing is performed, including visual segmentation, semantic analysis, and multi-agent collaborative operation.
It enables automated exploration testing of application pages in diverse scenarios, improving testing efficiency and accuracy, reducing testing and maintenance costs, and is suitable for personalized scenarios for different applications and users.
Smart Images
Figure CN122261995A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data testing technology, and in particular to a method and apparatus for page testing. Background Technology
[0002] Currently, mobile application testing primarily employs a fixed-logic execution approach, performing tests through pre-defined test elements and paths. This includes two main methods: 1) Control scripting: using tools like Appium (an open-source, cross-platform automated testing tool) to parse the application's view tree, relying on manually written scripts (using control IDs and XPath to locate elements); 2) Recording and playback: recording manual operation traces (including click coordinates and input content), which are then repeatedly executed during testing. However, this fixed-logic testing method heavily relies on manual operation and can only test fixed elements, resulting in generally poor test effectiveness and low efficiency. Summary of the Invention
[0003] In view of this, embodiments of the present invention provide a method and apparatus for page testing, which can improve testing efficiency and testing results.
[0004] To achieve the above objectives, according to one aspect of the present invention, a method for page testing is provided, comprising: Using the interactive page as the test page, the following test logic is executed on the test page to determine the test result of the interactive page: The page image of the page to be tested is divided into multiple segmentation images, and interactive elements in each segmentation image are identified to obtain a list of interactive elements of the page to be tested; the interactive elements in the list of interactive elements are tested to obtain the test results of the interactive elements; based on the test results of all interactive elements in the list of interactive elements, the test result of the interactive page is determined.
[0005] Optionally, the step of identifying interactive elements in each of the segmented images to obtain a list of interactive elements for the page under test includes: Identify the coordinates of candidate interactive elements in each of the segmented images, calculate the area of the candidate interactive elements based on their coordinates, and determine the list of interactive elements for the page to be tested based on the areas of the candidate interactive elements; and / or, Identify the text information of candidate interactive elements in each segmentation image; determine the list of interactive elements of the page to be tested based on the text information of the candidate interactive elements.
[0006] Optionally, determining the list of interactive elements of the page to be tested based on the text information of the candidate interactive elements includes: Semantic analysis is performed on the text information of the candidate interactive elements, and the business relevance between the candidate interactive elements and the page under test is calculated based on the semantic analysis results. The list of interactive elements for the page to be tested is determined based on the business relevance between the candidate interactive elements and the interactive page.
[0007] Optionally, before testing the interactive elements in the list of interactive elements, the method further includes: The interactive elements in the list of interactive elements are filtered according to a preset test breadth; and / or, The priority of the interactive element is determined based on its area, and the interactive elements in the interactive element list are sorted according to their priority.
[0008] Optionally, testing is performed on the interactive elements in the list of interactive elements, including: Attempt to retrieve the page corresponding to the interactive element in the list of interactive elements; In response to obtaining the jump page corresponding to the interactive element, the jump page is used as a new page to be tested, and the test logic is executed on the jump page to determine the test result of the interactive element; In response to the failure to obtain a redirect page corresponding to the interactive element, the test result of the interactive element is determined to be a redirection failure.
[0009] Optionally, attempting to retrieve the page corresponding to an interactive element in the list of interactive elements includes: Obtain the page image of the page obtained after performing a jump operation on the interactive elements in the list of interactive elements; determine the page similarity between the page image of the page and the page image of the interactive page; in response to the page similarity being less than a preset page similarity threshold, determine that no jump page corresponding to the interactive element has been obtained; in response to the page similarity being greater than or equal to the preset page similarity threshold, determine that the page is a jump page corresponding to the interactive element.
[0010] Optionally, the method further includes: Before executing the test logic on the redirected page, it is confirmed that the redirected page meets the preset iterative test conditions; the iterative test conditions include: the page level of the redirected page relative to the interactive page is less than the preset maximum page level, and / or the page content of the redirected page meets the preset page content conditions; If the redirected page does not meet the preset iterative test conditions, the test result of the interactive element is determined to be a successful redirect.
[0011] Optionally, the test results include: business logic verification results; the method further includes: In response to obtaining the jump page corresponding to the interactive element, the logical correlation between the jump page and the interactive page is determined; if the logical correlation is greater than or equal to a preset logical correlation threshold, the business logic verification result of the jump page is determined to be verified as passed; otherwise, the business logic verification result of the jump page is determined to be verified as failed.
[0012] According to another aspect of the present invention, a testing apparatus for an interactive page is provided, comprising: The testing module is used to perform the following test logic on the interactive page as the page to be tested, in order to determine the test result of the interactive page: The page image of the page to be tested is divided into multiple segmentation images, and interactive elements in each segmentation image are identified to obtain a list of interactive elements of the page to be tested; the interactive elements in the list of interactive elements are tested to obtain the test results of the interactive elements; based on the test results of all interactive elements in the list of interactive elements, the test result of the interactive page is determined.
[0013] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the interactive page testing method provided in the embodiments of the present invention.
[0014] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the testing method for the interactive page provided in the embodiments of the present invention.
[0015] According to another aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the testing method for the interactive page provided in the embodiments of the present invention.
[0016] One embodiment of the above invention has the following advantages or beneficial effects: it can automatically identify interactive elements in a page under test through page recognition, and then explore and test the page using the identified interactive elements. Specifically, dynamically identifying interactive elements in a page is applicable to diverse scenarios; for various pages, the elements to be tested can be determined through dynamic recognition. For example, in a personalized user experience scenario, interactive elements in interactive pages corresponding to different users can be obtained through dynamic recognition, tested, and accurate test results generated.
[0017] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description
[0018] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram illustrating the main steps of a page testing method according to an embodiment of the present invention; Figure 2 A flowchart of collaborative operation of multiple agents; Figure 3 This is a schematic diagram of the testing process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the page testing process when the preset test depth is 3 levels; Figure 5 A system architecture diagram for the testing methods applied to interactive pages; Figure 6 This is a schematic diagram of the main modules of a page testing apparatus according to an embodiment of the present invention; Figure 7 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation
[0019] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0020] It should be noted that the technical solutions disclosed in this invention, regarding the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.
[0021] With the increasing variety and number of mobile applications, the overall number of applications has also grown dramatically. To enable applications to provide better services, automated exploration testing is needed before deployment or after updates to evaluate their functionality. Existing testing methods primarily rely on fixed logic execution, requiring manual selection of test elements and paths based on the specific application. This approach is highly limiting, necessitates setting up extensive test data for different applications and scenarios, and results in extremely low testing efficiency. To address these shortcomings, this invention proposes a page testing method that automates application page exploration, intelligently identifies page elements, generates exploration paths, and performs tests. This significantly improves testing efficiency, avoids the complexity of manual operations, and reduces the difficulty of testing.
[0022] Figure 1 This is a schematic diagram of the main flow of a page testing method according to an embodiment of the present invention, such as... Figure 1 As shown, the page testing method mainly includes steps S101, S1011 to S1013.
[0023] Step S101: Using the interactive page as the page to be tested, execute the following test logic on the page to be tested to determine the test result of the interactive page: During testing, interactive pages within the application are acquired and designated as test pages for subsequent testing. For each test page, corresponding test logic is executed to automatically test the page and obtain the test results. These results include various aspects such as the presence of exceptions, successful redirection, and page accuracy; the specific information included can be flexibly configured according to actual needs.
[0024] The first interactive page encountered during testing is typically the application's initial page. A login process must be completed before testing this initial page to ensure the testing process proceeds smoothly. Subsequent interactive pages can be the pages accessed during each navigation, allowing for in-depth testing of these interactive pages.
[0025] Step S101 includes steps S1011 to S1013. The following logic needs to be executed for each interactive page to complete the page testing operation.
[0026] Step S1011: Divide the page image of the page to be tested into multiple segmentation images, identify the interactive elements in each segmentation image, and obtain a list of interactive elements of the page to be tested.
[0027] For the current page under test, the first step is to identify the elements within the page to determine the set of elements to be tested. Specifically, this can be done based on the page image. Before identification, the page image needs to be divided into multiple segments, each corresponding to an interactive element. By identifying the elements in each segment, a list of all interactive elements on the page under test can be obtained. This step can be achieved using the SAM algorithm, which automatically segments the page screenshot and outputs information about the interactive elements. This information includes coordinate information. The elements in the resulting list of interactive elements are the interactive elements on the current page under test that can be tested.
[0028] Step S1012: Test the interactive elements in the interactive element list to obtain the test results of the interactive elements.
[0029] The interactive elements in the list of interactive elements are tested, and the test results for each interactive element are obtained. This includes performing jump tests on interactive elements and determining the test results of interactive elements by judging whether the jump results meet expectations. The test results may also include whether triggering the jump to the interactive element was successful.
[0030] In one embodiment, when testing interactive elements in the list of interactive elements, it is possible to select only a portion of the interactive elements for testing, thereby reducing the amount of test data. When selecting interactive elements for testing, multiple interactive elements can be randomly selected based on a preset number, or elements located at the beginning of the page can be selected based on their coordinates, or other selection methods can be used, which can be flexibly set according to actual needs and scenarios.
[0031] In one embodiment, when testing each interactive element, the test path for that element can be a pre-set test path. That is, each interactive element on the page has a pre-set corresponding test path. The corresponding test path is selected based on the recognition result of the interactive element to perform the test and obtain the test result of the interactive element.
[0032] Step S1013: Determine the test results of the interactive page based on the test results of all interactive elements in the interactive element list.
[0033] After obtaining the test results for each interactive element in the list of interactive elements, the test results are merged to generate the complete test results for the final interactive page. In addition to the test results for each interactive element, the test results may also include analytical data generated based on the test results for each interactive element, such as anomaly rate and percentage of abnormal elements, to provide a more comprehensive display of the test results.
[0034] The page testing method provided by this invention can automatically identify interactive elements in a page under test through page recognition, and then explore and test the page using the identified interactive elements. The dynamic identification of interactive elements in a page is applicable to diverse scenarios; the elements to be tested can be determined dynamically for various types of pages. For example, in a personalized user experience scenario, interactive elements in interactive pages corresponding to different users can be obtained dynamically, tested, and accurate test results generated.
[0035] In one embodiment, identifying interactive elements in each segmentation image to obtain a list of interactive elements for the page under test includes: identifying the coordinates of candidate interactive elements in each segmentation image; calculating the area of the candidate interactive elements based on their coordinates; and determining the list of interactive elements for the page under test based on the area of the candidate interactive elements; and / or identifying the text information of the candidate interactive elements in each segmentation image; and determining the list of interactive elements for the page under test based on the text information of the candidate interactive elements.
[0036] When identifying each segmentation map, the information required for interactive elements includes the coordinates of the interactive elements. The area of the interactive element can be calculated from these coordinates. The calculated area is used to filter interactive elements in the list, eliminating those with areas too small (below a first preset threshold). The remaining interactive elements are then tested to reduce the amount of test data. The area of an interactive element can reflect its importance to some extent; therefore, more important interactive elements are prioritized for testing to ensure optimal results.
[0037] In addition to filtering based on the area of interactive elements, interactive elements can also be further filtered based on the text content they contain. For example, an AI visual reasoning model can be used to identify the text information contained in interactive elements in each segmentation image, and the importance of each interactive element can be determined based on the identification results, so that the elements in the list of interactive elements can be further sorted and filtered according to the determined importance.
[0038] In one embodiment, determining the list of interactive elements of the page to be tested based on the text information of candidate interactive elements includes: performing semantic analysis on the text information of candidate interactive elements, and calculating the business relevance between the candidate interactive elements and the page to be tested based on the semantic analysis results; and determining the list of interactive elements of the page to be tested based on the business relevance between the candidate interactive elements and the interactive page.
[0039] When determining the list of interactive elements, semantic analysis can be performed based on the obtained text information of the interactive elements to determine their functions, i.e., the business processes they are responsible for. For example, if the text information of an interactive element is "Click to sign in," analysis can determine that this interactive element is responsible for the "sign-in" business. Based on the semantic analysis results of each interactive element, the business relevance of each interactive element to the current page under test is determined. Then, the interactive elements with the highest business relevance can be selected to construct the interactive element list for the page under test. Business relevance refers to the degree of connection between the business content of an interactive element and the business of the current page. For example, for the homepage, interactive elements related to user sign-in and user login have a higher business relevance, while for pages after redirection, the business relevance of interactive elements related to user sign-in and user login may decrease. Based on the semantic analysis results and the coordinates of the interactive elements, the most important interactive elements on the current page can be clearly identified, and a list of interactive elements storing these most important interactive elements can be constructed to allow for subsequent testing of these important interactive elements and obtain accurate test results.
[0040] The page testing method provided by this invention can dynamically identify interactive elements in a page under test and determine a corresponding list of interactive elements for testing. Specifically, when determining the list of interactive elements, filtering can be performed based on element information and element coordinates to identify the interactive elements to be tested, thus generating an accurate list of interactive elements. Page testing is then performed based on these elements to obtain accurate test results and improve testing efficiency.
[0041] In one embodiment, before testing the interactive elements in the interactive element list, the method further includes: filtering the interactive elements in the interactive element list according to a preset test breadth; and / or determining the priority of the interactive elements based on the area of the interactive elements, and sorting the interactive elements in the interactive element list according to the priority of the interactive elements.
[0042] Before testing the list of interactive elements, it can be filtered according to preset test conditions. Since the list may contain a large number of interactive elements, to improve testing efficiency, it's necessary to select only a portion of them for testing. Specifically, the number of interactive elements to test on each page is determined according to a preset test breadth. For example, if the preset test breadth is 10, then only 10 elements need to be calculated for each page. Therefore, for each page, only 10 elements from its corresponding interactive element list are retained, representing the 10 elements of the interactive element page.
[0043] In addition to filtering the list of interactive elements, the elements in the list also need to be sorted. When sorting, the area of each interactive element is first determined, and then the priority of the interactive elements is determined based on their area. For example, the interactive element with the largest area has a priority of 1, which is the highest priority, and will be tested first. Besides area, business importance can also be considered; interactive elements responsible for core business functions will be given higher priority. The business importance of interactive elements can be identified through a large-scale model.
[0044] In one embodiment, the priority of interactive elements in the interactive element list can be dynamically maintained, that is, the priority of elements in the list is adjusted based on real-time generated data during testing. For example, during testing, the priority of elements in the list can be adjusted in real time based on the navigation result of the interactive element. Specifically, the priority of elements in the list that are of the same type as the current interactive element is adjusted based on the navigation result. For example, if the navigation result is a successful navigation and the correlation between the navigation page and the interactive element is greater than a first preset threshold, then the priority of elements of the same type as the current interactive element needs to be increased; similarly, if the navigation result is a failed navigation, or a successful navigation but the correlation between the navigation page and the interactive element is not greater than the first preset threshold, then the priority of elements of the same type as the current interactive element needs to be decreased.
[0045] The page testing method provided by this invention can prioritize identified interactive elements, enabling testing of important interactive elements to ensure comprehensiveness of the testing scope and generate more accurate test results. The priority setting allows the testing process to focus on more important elements. Through visual segmentation and semantic reasoning, the method can understand the functions of page elements, automatically associate component coordinates with semantic descriptions, and adapt to the application testing needs of different scenarios. No code modification or script writing is required; the method can seamlessly pass tests even when the application page is frequently adjusted, significantly reducing test maintenance costs and achieving a continuously automated testing process.
[0046] In one embodiment, testing interactive elements in a list of interactive elements includes: attempting to obtain a jump page corresponding to an interactive element in the list of interactive elements; in response to obtaining a jump page corresponding to an interactive element, using the jump page as a new page to be tested, executing test logic on the jump page to determine the test result of the interactive element; and in response to not obtaining a jump page corresponding to an interactive element, determining that the test result of the interactive element is a jump failure.
[0047] Testing interactive elements primarily includes redirection testing. This involves triggering the redirection behavior of interactive elements and determining whether the redirection was successful and whether a redirected page exists. If the redirection fails or no redirected page exists, a test result indicating a redirection failure is generated. If the redirection succeeds and a redirected page exists, this page is used as the page to be tested to execute test logic, complete the testing process for the redirected page, obtain the test result for the redirected page, and combine the test result of the redirected page with the current redirection result as the test result for the interactive elements on the current page.
[0048] The process involves identifying interactive elements on a page, obtaining the corresponding redirected pages, and then repeatedly identifying and testing these interactive elements on the redirected pages. This allows the system to autonomously generate an exploration path (test path) based on the interactive elements, delving deeper and deeper until the level of the obtained redirected pages meets the test conditions. It then returns to the list of interactive elements on the previous page, obtains the next interactive element, and tests it. This continues until the number of interactive elements on the previous page meets the preset test breadth. Finally, it returns to the page one level above the previous page, and so on, to complete the testing of all interactive elements and obtain the final test results.
[0049] In one embodiment, attempting to obtain the jump page corresponding to an interactive element in the list of interactive elements includes: Get the page image of the page obtained after performing a jump operation on an interactive element in the list of interactive elements; determine the page similarity between the page image of the page and the page image of the interactive page; if the page similarity is less than a preset page similarity threshold, determine that no jump page corresponding to the interactive element has been obtained; if the page similarity is greater than or equal to the preset page similarity threshold, determine that the page is the jump page corresponding to the interactive element.
[0050] When attempting to retrieve the page corresponding to an interactive element, the process first performs a jump operation on the interactive element and obtains the jump result. Only when the jump result contains information indicating a successful jump can the corresponding page image of the redirected page be obtained. However, the redirected page at this point may not be the correct page, i.e., it may not be the page corresponding to the interactive element. For example, clicking the "Check-in" element may not return the current page, indicating a possible logical error. Therefore, for the redirected page, its similarity to the interactive page needs to be calculated. This can be done by comparing screenshots of the two pages. Only when the similarity calculation result is greater than or equal to a preset page similarity threshold is the jump considered successful; otherwise, the jump fails. The similarity calculation uses the SSIM (Structural Similarity) algorithm. In this algorithm, a larger result indicates less similarity, and a smaller result indicates greater similarity. Therefore, a result greater than or equal to the preset page similarity threshold indicates that the two pages before and after the jump are dissimilar, and the current jump operation is valid. If other algorithms are used to calculate similarity, and a higher similarity result indicates greater similarity, then a condition needs to be set such that if the similarity is greater than or equal to a preset similarity threshold, the jump is deemed invalid.
[0051] The page testing method provided by the present invention can perform jump tests based on dynamically identified interactive elements, and complete the testing of interactive elements by verifying the jump results. This realizes a testing method that automatically explores test paths based on dynamically identified interactive elements, which greatly improves testing efficiency.
[0052] In one embodiment, the method further includes: before executing test logic on the redirected page, confirming that the redirected page meets preset iterative test conditions; the iterative test conditions include: the page level of the redirected page relative to the interactive page is less than a preset maximum page level, and / or the page content of the redirected page meets preset page content conditions; in response to the redirected page not meeting the preset iterative test conditions, determining that the test result of the interactive element is a successful redirect.
[0053] Before testing the redirected page, it needs to be validated to determine if it meets the iterative testing conditions, which in turn determines whether further in-depth testing is necessary. Testing will only continue on the redirected page if the iterative testing conditions are met; otherwise, the test ends, and a successful redirection result is generated. The iterative conditions include: the redirected page's page level is less than the preset maximum page level (meaning the current test depth has not reached the preset maximum depth and further in-depth testing is required); and / or the redirected page's page content meets preset page content conditions (meaning the redirection result is considered valid and testing should continue).
[0054] In one embodiment, the test result includes: business logic verification result; the method further includes: In response to obtaining the redirect page corresponding to the interactive element, determine the logical correlation between the redirect page and the interactive page; if the logical correlation is greater than or equal to the preset logical correlation threshold, determine that the business logic verification result of the redirect page is verified as passed; otherwise, determine that the business logic verification result of the redirect page is verified as failed.
[0055] Before testing the redirected page, its accuracy needs to be verified to determine if there is a correlation between it and the clicked interactive elements, thus confirming whether the redirection operation led to the correct page. Specifically, the logical correlation between the redirected page and the interactive page is calculated. This correlation is determined by analyzing the text information of the identified interactive elements. Semantic analysis is performed on the text information of the interactive elements to obtain the analysis results, and the correlation between these results and the redirected page is then assessed. For example, if the analysis result indicates that the interactive element is a check-in button, but the redirected page is an points page, the correlation between the two is low, below the preset logical correlation threshold, and the verification fails. Otherwise, the verification passes, and the result is recorded.
[0056] After determining the relevance, the priority of elements in the interactive element list on the interactive page can be dynamically adjusted based on the relevance. If the verification result is passed, the priority of similar elements is increased; if the verification result is failed, the priority of similar elements is decreased.
[0057] The page testing method provided by this invention can dynamically identify interactive elements contained in the page under test, obtain the elements that need to be tested on the page, and automatically explore and generate test paths based on the identified interactive elements through preset test depth and automatic element jump operations. The generated test paths are used to perform automated testing on the page under test to obtain test results. This method breaks through scenario limitations and can dynamically identify interactive elements contained in pages of different scenarios and applications to achieve testing of interactive elements, thus improving testing efficiency. Specifically, when testing interactive elements, corresponding test results can be generated based on the jump results. Simultaneously, the correlation between interactive elements and jump pages can be verified to ensure the correctness of jump pages during testing, thereby achieving testing of business logic. This improves the accuracy of test results and solves the problem of being unable to verify business logic by relying on fixed test logic.
[0058] In one embodiment, the entire testing process can be completed through the collaborative operation of multiple intelligent agents, including a perceptual agent, an execution agent, and a verification agent. The perceptual agent is responsible for identifying interactive elements on the page and performing semantic analysis of the text information of these interactive elements; the execution agent is responsible for executing click operations on the interactive elements, or operations such as selection to trigger navigation to the interactive elements; and the verification agent is responsible for testing and verifying the navigation results of the interactive elements. Figure 2 The diagram illustrates the collaborative operation flowchart of multiple agents. The perceptual agent segments and identifies interactive elements from the page image under test. This information is then input into the executing agent, which performs click operations on these interactive elements. For successfully navigated pages, the page is tested. This involves the perceptual agent identifying interactive elements and the executing agent performing click operations on them, continuing until the page reaches a preset depth. For pages that fail to navigate, the verification agent verifies the result, generating test data for the failed navigation and backtracking to the executing agent to perform click operations on the next interactive element. This collaborative testing architecture of multiple agents enables a dynamic closed loop for automated exploration and testing logic, achieving automated page exploration and testing. Perception Agent: The SAM (Segment Anything Model) model performs pixel-level segmentation of the page, accurately locating the coordinates of all interactive elements. It combines AI (Artificial Intelligence) visual reasoning to analyze the semantic functions of elements and calculates element priorities based on rules, outputting a list of high-value exploration targets. Execution Agent: Based on the perception results, it dynamically plans exploration paths, triggering new page exploration when changes occur and switching elements when no changes occur; it also supports automatic backtracking. Verification Agent: It uses the EfficientNet (Compound Scaling Convolutional Neural Network) model to quickly identify explicit anomalies such as blank screens and error pages; it generates "trigger element descriptions" and "new page descriptions" through AI visual reasoning, calculates logical correlation, and identifies logical mismatches between "elements" and "pages."
[0059] Figure 3 This is a schematic diagram of the test process according to an embodiment of the present invention. Figure 3As shown, after the test begins, task initialization is performed first, including checking the device and obtaining the configured test tasks. The test task configuration information is parsed, including obtaining the list of pages to be tested, the preset test breadth, task execution time, preset test depth, and test account information. Initialization operations are completed by the execution agent, including clearing data in the application, opening the application, completing login, opening the initial page, and taking a screenshot. The initial page is verified by the verification agent, including image classification of the initial page, analysis of the page based on prompts, and recording the page display results. The initial page is identified by the perception agent, including identifying the coordinates of interactive elements using the SAM algorithm, identifying interactive element information based on prompts, and recording the coordinates and information of interactive elements. A page exploration operation is then performed, i.e., a depth test. The execution agent clicks on the current page element, obtaining a screenshot of the page after the click. It is determined whether the pages before and after the click are similar; if they are similar, it indicates an invalid jump, and the execution agent returns to perform the click operation on the next interactive element. If there is no similarity, the jump is considered valid. Verification is then performed using a verification agent, including identifying and analyzing the displayed page, verifying the correlation between the jump page and interactive elements, and calculating the identification, analysis, and verification results. It is then determined whether the current jump page has reached the preset test depth. If so, the process returns to the previous page to execute the click operation on the next interactive element, until the number of tested interactive elements on the previous page reaches the preset test breadth. Then, the process returns to the page before that, until the number of tested interactive elements across all levels of pages reaches the preset test breadth. If the preset test depth has not been reached, the perception agent continues to identify the coordinates and information of interactive elements on the jump page, and then returns to the previous page for further exploration.
[0060] Figure 4 This is a schematic diagram of the page testing process when the preset test depth is 3 levels. The example uses a preset test breadth of 3 and a preset test depth of 3. The jump relationships during testing are as follows: First, based on the interactive elements of the initial page, all elements are traversed, and clicking triggers a jump operation. At this point, the page jumps to the level 1 page. The jump relationship is recorded, and a queue of the top 3 elements of the level 1 page is generated. Then, clicking the top 3 elements of the level 1 page in sequence triggers a jump to the level 2 page. The jump relationship is recorded, and a queue of the top 3 elements of the level 2 page is generated. Then, clicking the top 3 elements of the level 2 page in sequence triggers a jump to the level 3 page. At this point, the preset test depth is reached, and no further jumps are made. After completing the test on the level 3 page, the page returns to the level 2 page. After the top 3 elements of the level 2 page are clicked, the page returns to the level 1 page. After the top 3 elements of the level 1 page are clicked, the page returns to the initial page. The process of traversing the next element on the initial page is repeated until all elements have been traversed.
[0061] The executing agent retrieves the initial page (S0) element list from the hierarchical stack and selects the highest priority element (such as the "Check-in" button): First, it captures a screenshot of the current page. It performs a click operation, waits for the page to stabilize, and then captures a screenshot of the operation (denoted as S1, the check-in page). The executing agent calls the SSIM algorithm to compare the screenshots before and after the operation, obtaining the SSIM result. If the result is less than a certain threshold, it is considered that the page may not have redirected, indicating a "click-without-response" problem, and switches to clicking the next element on page S0; if the result is greater than or equal to a certain threshold, it is considered a "valid redirect." The verifying agent handles S1: The verifying agent performs explicit anomaly detection on S1: It calls the EfficientNet model to determine if S1 is a "normal page"; if an anomaly is detected, it needs to be marked as an error type, reverts to S0, and clicks the next element on page S0; if it is determined to be a "normal page," the logical correlation between the "Check-in" button and S1 (e.g., 0.92) is greater than or equal to a threshold (e.g., 0.7, configurable as needed), indicating "logically normal"; the result is fed back to the perceptual agent, increasing the priority of similar elements. Since the current level 1 is less than the exploration depth 3, the perception agent performs the S1 duplicate element identification process, generates a check-in page element list and pushes it onto the level stack (the top of the stack is the check-in page, level 1), and the execution agent continues to explore down to the next page (level 2). When the exploration reaches the maximum depth (level 3), the execution agent triggers backtracking: the top page of the stack is popped out, and the agent goes back to the previous page via ADB; the execution agent selects high-priority elements that have not been operated on and continues to explore until the number of elements explored on that page reaches the exploration breadth (10); the agent goes back to the homepage and explores other high-priority elements until the homepage elements have been explored.
[0062] Figure 5 This is a system architecture diagram for testing methods on interactive pages. The system comprises a multi-layered structure consisting of task management, intelligent agents, a model service layer, a knowledge base, and device control. Task management includes test scenario configuration, parameter management, scheduled tasks, and alarm configuration, used to configure relevant parameters for test tasks. Intelligent agents include created perceptual agents, execution agents, and verification agents. The model service layer includes multiple models involved in the testing process, such as the SAM image segmentation model, AI semantic analysis model, AI visual reasoning model, SSIM model, and EffficientNet classification model. The knowledge base includes an element feature analysis library (for element identification), an application page screenshot library, and an interaction anomaly case library (for verifying the correctness of page navigation). Device control includes controlling operations such as element clicks, text input, application installation / uninstallation, and screenshots.
[0063] Figure 6 The diagram shows the main modules of a testing device for an interactive page according to an embodiment of the present invention. Figure 6As shown, the interactive page testing device 600 mainly includes a first testing module 601, an identification module 6011, a second testing module 6012, and a determination module 6013.
[0064] The first test module 601 is used to execute the following test logic on the interactive page as the page to be tested, in order to determine the test results of the interactive page: The recognition module 6011 is used to divide the page image of the page to be tested into multiple segmentation images, identify interactive elements in each segmentation image, and obtain a list of interactive elements of the page to be tested; the second testing module 6012 is used to test the interactive elements in the list of interactive elements and obtain the test results of the interactive elements; the determination module 6013 is used to determine the test results of the interactive page based on the test results of all interactive elements in the list of interactive elements.
[0065] The page testing apparatus provided in this embodiment of the invention can automatically identify interactive elements in a page under test through page recognition, and then explore and test the page using the identified interactive elements. The dynamic identification of interactive elements in a page is applicable to diverse scenarios; the elements to be tested can be determined dynamically for various types of pages. For example, in a personalized user experience scenario, interactive elements in interactive pages corresponding to different users can be obtained dynamically, tested, and accurate test results generated.
[0066] In one embodiment, the identification module 6011 is configured to: identify the coordinates of candidate interactive elements in each segmentation image; calculate the area of the candidate interactive elements based on the coordinates of the candidate interactive elements; determine the list of interactive elements of the page to be tested based on the area of the candidate interactive elements; and / or, identify the text information of the candidate interactive elements in each segmentation image; and determine the list of interactive elements of the page to be tested based on the text information of the candidate interactive elements.
[0067] In one embodiment, the identification module 6011 is configured to: perform semantic analysis on the text information of candidate interactive elements, and calculate the business relevance between the candidate interactive elements and the page under test based on the semantic analysis results; and determine the list of interactive elements of the page under test based on the business relevance between the candidate interactive elements and the interactive page.
[0068] In one embodiment, the testing device 600 for the interactive page further includes: a filtering module 6014 (not shown in the figure), used to filter interactive elements in the interactive element list according to a preset test breadth; and / or a sorting module 6015 (not shown in the figure), used to determine the priority of interactive elements based on the area of interactive elements, and sort the interactive elements in the interactive element list according to the priority of interactive elements.
[0069] In one embodiment, the second testing module 6012 is configured to: attempt to obtain a jump page corresponding to an interactive element in the list of interactive elements; in response to obtaining a jump page corresponding to an interactive element, use the jump page as a new page to be tested, execute test logic on the jump page to determine the test result of the interactive element; and in response to not obtaining a jump page corresponding to an interactive element, determine that the test result of the interactive element is a jump failure.
[0070] In one embodiment, the second testing module 6012 is configured to: acquire a page image of the page obtained after performing a jump operation on an interactive element in the list of interactive elements; determine the page similarity between the page image of the page and the page image of the interactive page; determine that no jump page corresponding to the interactive element has been acquired in response to the page similarity being less than a preset page similarity threshold; and determine that the page is a jump page corresponding to the interactive element in response to the page similarity being greater than or equal to the preset page similarity threshold.
[0071] In one embodiment, the determining module 6013 is used to confirm that the redirected page meets preset iterative test conditions before executing test logic on the redirected page; the iterative test conditions include: the page level of the redirected page relative to the interactive page is less than the preset maximum page level, and / or the page content of the redirected page meets the preset page content conditions; in response to the redirected page not meeting the preset iterative test conditions, the test result of the interactive element is determined to be a successful redirect.
[0072] In one embodiment, the test results include: business logic verification results; a determination module 6013, configured to determine the logical correlation between the jump page and the interactive page in response to obtaining the jump page corresponding to the interactive element; if the logical correlation is greater than or equal to a preset logical correlation threshold, the business logic verification result of the jump page is determined to be verified as passed; otherwise, the business logic verification result of the jump page is determined to be verified as failed.
[0073] The page testing apparatus provided by this invention can dynamically identify interactive elements contained in a page under test, obtain the elements to be tested on the page, and automatically explore and generate test paths based on the identified interactive elements through preset test depth and automatic element jump operations. The generated test paths are used to automatically test the page under test and obtain test results. This breaks through scenario limitations, dynamically identifying interactive elements contained in pages of different scenarios and applications to achieve testing of interactive elements, thus improving testing efficiency. Specifically, when testing interactive elements, corresponding test results can be generated based on the jump results. Simultaneously, the correlation between interactive elements and jump pages can be verified to ensure the correctness of jump pages during testing, achieving testing of business logic and improving the accuracy of test results while solving the problem of being unable to verify business logic by relying on fixed test logic.
[0074] Figure 7 An exemplary system architecture 700 is shown, in which the page testing method or apparatus of embodiments of the present invention can be applied.
[0075] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, and 703, a network 704, and a server 705. Network 704 serves as the medium for providing communication links between terminal devices 701, 702, and 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.
[0076] Users can use terminal devices 701, 702, and 703 to interact with server 705 via network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).
[0077] Terminal devices 701, 702, and 703 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.
[0078] Server 705 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 701, 702, and 703 (for example only). The backend management server can analyze and process data such as received page test requests, and feed back the processing results (such as test results - for example only) to the terminal devices.
[0079] It should be noted that the page testing method provided in this embodiment of the invention is generally executed by server 705, and correspondingly, the page testing device is generally set in server 705.
[0080] It should be understood that Figure 7 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0081] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing terminal devices or servers of the present invention. Figure 8 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.
[0082] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0083] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.
[0084] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this invention.
[0085] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0086] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0087] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including a first testing module, an identification module, a second testing module, and a determination module. The names of these units or modules do not necessarily limit the specific unit or module itself. For example, the first testing module can also be described as "a module for using an interactive page as the page to be tested, executing the following test logic on the page to be tested, in order to determine the test result of the interactive page."
[0088] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: Using the interactive page as the page to be tested, execute the following test logic on the page to determine the test results of the interactive page: The page image of the page to be tested is divided into multiple segment images, and interactive elements in each segment image are identified to obtain a list of interactive elements of the page to be tested. The interactive elements in the list of interactive elements are tested to obtain the test results of the interactive elements. Based on the test results of all interactive elements in the list of interactive elements, the test result of the interactive page is determined.
[0089] According to the technical solution of this invention, interactive elements in a test page can be automatically identified through page recognition, allowing for exploration and testing of the page using these identified interactive elements. Dynamically identifying interactive elements in a page is applicable to diverse scenarios; the elements to be tested can be determined dynamically for various types of pages. For example, in a personalized user experience scenario, dynamic identification can be used to obtain interactive elements in the interactive pages corresponding to different users, conduct tests, and generate accurate test results.
[0090] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A method for page testing, characterized in that, include: Using the interactive page as the test page, the following test logic is executed on the test page to determine the test result of the interactive page: The page image of the page to be tested is divided into multiple segmentation images, and the interactive elements in each segmentation image are identified to obtain a list of interactive elements of the page to be tested. Test the interactive elements in the list of interactive elements to obtain the test results of the interactive elements; The test results of the interactive page are determined based on the test results of all interactive elements in the list of interactive elements.
2. The method according to claim 1, characterized in that, The step of identifying interactive elements in each segmentation map to obtain a list of interactive elements for the page under test includes: Identify the coordinates of candidate interactive elements in each of the segmented images, calculate the area of the candidate interactive elements based on their coordinates, and determine the list of interactive elements for the page to be tested based on the areas of the candidate interactive elements; and / or, Identify the text information of candidate interactive elements in each segmentation image; determine the list of interactive elements of the page to be tested based on the text information of the candidate interactive elements.
3. The method according to claim 2, characterized in that, Determining the list of interactive elements of the page to be tested based on the text information of the candidate interactive elements includes: Semantic analysis is performed on the text information of the candidate interactive elements, and the business relevance between the candidate interactive elements and the page under test is calculated based on the semantic analysis results. The list of interactive elements for the page to be tested is determined based on the business relevance between the candidate interactive elements and the interactive page.
4. The method according to any one of claims 1-3, characterized in that, Before testing the interactive elements in the list of interactive elements, the method further includes: The interactive elements in the list of interactive elements are filtered according to a preset test breadth; and / or, The priority of the interactive element is determined based on its area, and the interactive elements in the interactive element list are sorted according to their priority.
5. The method according to claim 1, characterized in that, Testing the interactive elements in the list of interactive elements includes: Attempt to retrieve the page corresponding to the interactive element in the list of interactive elements; In response to obtaining the jump page corresponding to the interactive element, the jump page is used as a new page to be tested, and the test logic is executed on the jump page to determine the test result of the interactive element; In response to the failure to obtain a redirect page corresponding to the interactive element, the test result of the interactive element is determined to be a redirection failure.
6. The method according to claim 5, characterized in that, Attempt to retrieve the page corresponding to the interactive element in the list of interactive elements, including: Obtain the page image of the page obtained after performing a jump operation on the interactive elements in the list of interactive elements; determine the page similarity between the page image of the page and the page image of the interactive page; in response to the page similarity being less than a preset page similarity threshold, determine that no jump page corresponding to the interactive element has been obtained; in response to the page similarity being greater than or equal to the preset page similarity threshold, determine that the page is a jump page corresponding to the interactive element.
7. The method according to claim 5, characterized in that, The method further includes: Before executing the test logic on the redirected page, it is confirmed that the redirected page meets the preset iterative test conditions; the iterative test conditions include: the page level of the redirected page relative to the interactive page is less than the preset maximum page level, and / or the page content of the redirected page meets the preset page content conditions; If the redirected page does not meet the preset iterative test conditions, the test result of the interactive element is determined to be a successful redirect.
8. The method according to any one of claims 5-7, characterized in that, The test results include: business logic verification results; the method further includes: In response to obtaining the jump page corresponding to the interactive element, the logical correlation between the jump page and the interactive page is determined; if the logical correlation is greater than or equal to a preset logical correlation threshold, the business logic verification result of the jump page is determined to be verified as passed; otherwise, the business logic verification result of the jump page is determined to be verified as failed.
9. A testing device for an interactive page, characterized in that, include: The first testing module is used to perform the following test logic on the interactive page as the page to be tested, in order to determine the test result of the interactive page: The recognition module is used to divide the page image of the page to be tested into multiple segmentation images, identify interactive elements in each segmentation image, and obtain a list of interactive elements of the page to be tested. The second testing module is used to test the interactive elements in the interactive element list and obtain the test results of the interactive elements. The determination module is used to determine the test results of the interactive page based on the test results of all interactive elements in the list of interactive elements.
10. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.
11. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.
12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.