Automated testing method and related apparatus

Abstract

The application discloses an automatic testing method and related device, the method comprises the following steps: obtaining the page visual feature of a test page; obtaining a pre-recorded test script; wherein the test script comprises a preset visual feature and a first operation corresponding to the preset visual feature; matching the page visual feature with the preset visual feature to determine a matching result; if the matching result meets a preset matching condition, executing the first operation corresponding to the preset visual feature; if the matching result does not meet the preset matching condition, calling a visual language model to process the test page, and executing a second operation according to the processing result of the visual language model. The application realizes directly executing the pre-recorded test script when the current page matches the pre-recorded page, and re-judging and executing the operation by the visual language model when the current page does not match the pre-recorded page, thereby reducing the script execution failure caused by page changes and improving the stability of automatic testing.

Description

Technical Field

[0001] This application relates to the field of automated testing technology, and in particular to automated testing methods and related apparatus. Background Technology

[0002] In automated testing scenarios, in order to analyze the application page state and complete repetitive operations, automated scripts are usually needed to identify controls, pop-ups, or abnormal states on the current page, thereby determining the subsequent operation paths such as clicks, inputs, or jumps.

[0003] In related technologies, automated testing is typically based on preset rules. It uses Optical Character Recognition (OCR) to analyze text or locate elements on the page to determine the current page state and execute predefined operations. Alternatively, it can combine a Vision-Language Model (VLM) to first understand the page and record the operation path, then replay this path in subsequent tests for automated testing. However, if changes occur on the page during the playback of the operation path that were not observed during the recording phase, a mismatch between the recorded operation path and the actual page state can occur, thus affecting the stability of the automated test execution. Summary of the Invention

[0004] In view of the above problems, this application provides an automated testing method and related apparatus to improve the execution stability of automated testing when unexpected changes occur to the page, while also considering testing cost efficiency. The specific solution is as follows:

[0005] The first aspect of this application provides an automated testing method, comprising:

[0006] Obtain the visual features of the test page;

[0007] Obtain a pre-recorded test script; wherein the test script includes preset visual features and a first operation corresponding to the preset visual features;

[0008] The page visual features are matched with the preset visual features to determine the matching result;

[0009] If the matching result meets the preset matching conditions, execute the first operation in the test script corresponding to the preset visual feature;

[0010] If the matching result does not meet the preset matching conditions, the visual language model is invoked to process the test page, and a second operation is performed based on the processing result of the visual language model.

[0011] Optionally, obtaining the visual features of the test page includes:

[0012] Capture the page image of the test page;

[0013] The page image is subjected to discrete cosine transform using the perceptual hash (pHash) algorithm to extract the low-frequency components of the page image;

[0014] The low-frequency components are encoded into hash values, which are used as the visual features of the test page.

[0015] Optionally, matching the page visual features with the preset visual features to determine the matching result includes:

[0016] Determine the feature difference value between the page visual features and the preset visual features;

[0017] The feature difference value is compared with a preset difference threshold, and the matching result is determined based on the comparison result.

[0018] Optionally, determining the feature difference value between the page visual features and the preset visual features includes:

[0019] Obtain the first hash value corresponding to the page visual feature and the second hash value corresponding to the preset visual feature;

[0020] Compare the first hash value and the second hash value bit by bit;

[0021] The number of bits that differ in value between the first hash value and the second hash value is counted and determined as the feature difference value between the page visual feature and the preset visual feature.

[0022] Optionally, the preset difference threshold is determined based on the page change frequency of the test page; wherein, when the page change frequency is less than a preset frequency threshold, a first difference threshold is determined as the preset difference threshold; when the page change frequency is greater than or equal to the preset frequency threshold, a second difference threshold is determined as the preset difference threshold; the second difference threshold is greater than the first difference threshold.

[0023] Optionally, the first operation includes at least one of a click operation, a swipe operation, and an input operation; the step of executing the first operation in the test script corresponding to the preset visual feature if the matching result meets the preset matching conditions includes:

[0024] If the matching result meets the preset matching condition, obtain the operation coordinates and operation type corresponding to the preset visual feature;

[0025] The first operation is performed on the test page according to the operation coordinates and the operation type.

[0026] Optionally, if the matching result does not meet the preset matching conditions, the visual language model is invoked to process the test page, and a second operation is performed based on the processing result of the visual language model, including:

[0027] If the matching result does not meet the preset matching conditions, a preset prompt message is constructed; wherein, the preset prompt message includes at least an operation instruction description and a desired output format, the operation instruction description is used to instruct the visual language model to identify the page elements to be processed in the test page and output the target page operation information, and the desired output format is used to constrain the visual language model to return structured target page operation information; the target page operation information includes element position coordinates and target page operation type;

[0028] The collected page image of the test page and the preset prompt information are input into the visual language model;

[0029] Receive the target page operation information output by the visual language model, and parse the target page operation information to obtain the element position coordinates and the target page operation type;

[0030] Based on the element's position coordinates and the target page's operation type, the second operation is performed on the test page.

[0031] Optionally, after performing the second operation based on the processing result of the visual language model, the method further includes:

[0032] Record the operation coordinates, operation type, and operation sequence corresponding to the second operation;

[0033] Establish the correspondence between the visual features of the test page and the second operation;

[0034] The test script is updated based on the page's visual features and the second operation corresponding to those features.

[0035] A second aspect of this application provides an automated testing apparatus, comprising:

[0036] The feature acquisition unit is used to acquire the visual features of the test page;

[0037] A script acquisition unit is used to acquire a pre-recorded test script; wherein the test script includes preset visual features and a first operation corresponding to the preset visual features;

[0038] The matching determination unit is used to match the page visual features with the preset visual features and determine the matching result;

[0039] The first execution unit is configured to perform a first operation corresponding to the preset visual feature if the matching result meets the preset matching conditions.

[0040] The second execution unit is used to call the visual language model to process the test page if the matching result does not meet the preset matching conditions, and to perform a second operation based on the processing result of the visual language model.

[0041] A third aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0042] The memory is used to store computer programs;

[0043] The processor is used to execute the computer program so that the electronic device can implement the automated testing method of the first aspect or any implementation thereof.

[0044] A fourth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to perform an automated testing method as described in the first aspect or any implementation thereof.

[0045] The fifth aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the automated testing method of the first aspect or any implementation thereof.

[0046] By employing the above technical solution, this application records not only the first operation but also the preset visual features corresponding to the first operation in the test script. During test execution, the visual features of the test page are acquired, and the current test page's visual features are matched with the preset visual features recorded in the test script to determine whether the current test page is still suitable for executing the first operation. This avoids mechanically applying recorded operations when the test page state changes, reducing the problem of automated test execution failures caused by mismatches between recorded operations and actual page states, and improving the stability of automated test execution. If the matching result meets the preset matching conditions, the first operation corresponding to the preset visual features in the test script is directly reused, reducing the number of times the visual language model is called in repetitive test processes, thereby reducing the operating cost of automated testing and improving test execution efficiency. If the matching result does not meet the preset matching conditions, that is, when the current test page is no longer suitable for the recorded first operation, the visual language model is called to process the test page, and the second operation is executed based on the processing result. Therefore, this application reduces the cost of repeated calls to the video language model while retaining the visual language model's ability to handle page changes, pop-up interference, and other situations, thus balancing the cost of automated testing with fault tolerance and improving the stability and adaptability of the entire automated testing method. Attached Figure Description

[0047] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0048] Figure 1 A schematic diagram of a system architecture provided for an embodiment of this application;

[0049] Figure 2 This application provides a schematic diagram of a conventional automated script execution process.

[0050] Figure 3 A flowchart illustrating an automated testing process using OCR technology, provided as an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of an automated testing method provided in an embodiment of this application;

[0052] Figure 5 This is a schematic diagram of an automated testing device provided in an embodiment of this application;

[0053] Figure 6 This is a schematic diagram of a computer device structure provided in an embodiment of this application. Detailed Implementation

[0054] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the Description of Embodiments section is only for explaining specific embodiments of this application and is not intended to limit this application. The terms "first," "second," etc., in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0055] The method provided in this application embodiment can be applied to, for example, Figure 1 In the system architecture shown, terminal 102 communicates with server 101 via a network, and data storage system 100 can store data that server 101 needs to process or requires. Data storage system 100 can be integrated onto server 101 or deployed in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, such as smartwatches, smart bracelets, and head-mounted devices. Server 101 can be implemented by a standalone server or by a server cluster consisting of multiple servers. In this embodiment, data storage system 100 can be a standalone physical storage server, a distributed storage system consisting of multiple storage nodes, or a cloud storage system providing services such as object storage, relational databases, cache databases, cloud databases, and cloud storage. For example, in large-scale automated testing applications, a distributed storage system can be used to save test scripts, page images, page visual features, execution logs, and test results, and multiple test tasks can be scheduled in parallel through a server cluster. In scenarios where visual language models need to be invoked to handle page changes, page image understanding and operation information generation can be completed through cloud-based model services or local model services to improve the adaptability of automated testing processes.

[0056] Terminal 102 can serve as the execution end for automated testing tasks, running test pages of the application under test. During testing, it captures page images, performs page operations such as clicks, swipes, and inputs, and provides feedback on the execution status of the automated test. Terminal 102 can also obtain the visual features of the test page based on the test scripts issued by server 101, and transmit the visual features, page images, operation execution results, and abnormal page information to server 101 via the network. Terminal 102 can also have local preprocessing capabilities, such as taking screenshots of the test page, extracting page visual features, performing matching between page visual features and preset visual features, or directly executing the first operation based on the operation coordinates and operation type in the test script, thereby reducing the computational burden on the server side. Furthermore, terminal 102 can provide a graphical user interface, allowing testers to create test tasks, select the application under test, view the test script recording process, view the automated test execution results, or manually confirm the page operation information output by the visual language model, facilitating the configuration, execution, and result analysis of automated testing tasks.

[0057] Server 101 may be deployed with a script management module, a feature matching module, a test scheduling module, and a visual language model processing module, which are respectively used to manage test scripts, execute page visual feature matching, schedule automated test tasks, and call the visual language model for page processing when the page visual features do not match the preset visual features in the script. The script management module in server 101 can store and maintain pre-recorded test scripts, which include preset visual features and a first operation corresponding to the preset visual features. The test scheduling module can send test tasks and test scripts to terminal 102 and receive the page visual features or page images of the test page returned by terminal 102. The feature matching module can match the page visual features of the test page with the preset visual features in the test script to determine the matching result. When the matching result meets the preset matching conditions, server 101 can instruct terminal 102 to execute the first operation in the test script corresponding to the preset visual features; when the matching result does not meet the preset matching conditions, the visual language model processing module can process the page image and preset prompt information of the test page, output target page operation information, so that terminal 102 can execute a second operation according to the target page operation information.

[0058] In some implementations, server 101 can also be used for recording and updating test scripts. During the test preparation phase, server 101 can call a visual language model to process the page images captured by terminal 102, determine the first operation to be performed on the current page, and record the operation coordinates, operation type, and operation order corresponding to this operation. Simultaneously, it records the preset visual features of the page before the operation is executed, thereby generating a test script that includes the correspondence between the preset visual features and the first operation. During the test execution phase, when the visual language model generates and executes a second operation for a test page that does not meet the preset matching conditions, server 101 can also record the operation coordinates, operation type, and operation order corresponding to the second operation, and establish the correspondence between the page visual features of the test page and the second operation, thereby updating the test script. In this way, the test script can be expanded as new page scenarios are encountered during test execution, reducing the repeated calls to the visual language model when encountering the same or similar page scenarios again.

[0059] It should be noted that the method provided in this application can be executed independently on the server side, independently on the terminal device side, or collaboratively by the terminal device and the server. For example, for terminal devices with strong computing capabilities, page image acquisition, page visual feature extraction, feature matching, test script execution, and visual language model inference can be completed locally on terminal 102. For terminal devices with limited computing resources, terminal 102 can be responsible for running the application under test, acquiring page images, and performing page operations, while server 101 can handle page visual feature extraction, feature matching, test script management, and visual language model processing. Furthermore, in large-scale compatibility testing, performance automation testing, or multi-device concurrent testing scenarios, server 101 can uniformly schedule multiple terminals 102 to execute test tasks and centrally store test scripts, page visual features, operation records, and test results, thereby improving the management and execution efficiency of automated test tasks.

[0060] Figure 1 The system architecture of the illustrated embodiments is provided to more clearly illustrate the technical solutions of the embodiments of this application and does not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will recognize that, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar automated testing problems. For example, the method of this application can be applied to scenarios such as mobile application automated testing, web page automated testing, mini-program automated testing, client performance automated testing, regression testing, and multi-version compatibility testing. Furthermore, the storage location of test scripts, the deployment method of visual language models, the extraction location of page visual features, and the scheduling method of test tasks can be flexibly configured according to business needs, thereby achieving more adaptable and scalable automated testing services.

[0061] Before introducing the automated testing method provided in the embodiments of this application, the automated testing methods in related technologies will be described first. (Refer to...) Figure 2 Automated scripts in related technologies typically employ a process flow of preset conditions, condition judgments, and execution operations. Specifically, script developers need to pre-set the conditions required to enter a certain page scenario based on the business flow of the application or page under test, and determine whether the current page meets these conditions during script execution. If the conditions are met, the corresponding automated operation is executed, such as clicking, inputting, or redirecting. If the conditions are not met, a preset error handling process is entered, or the issue is handled according to pre-configured exception handling logic. It is evident that the above method can run stably when the page flow and exception conditions are fully preset. However, it heavily relies on the script developer's understanding of the automated testing scenario. When pop-ups, overlays, control changes, or page state changes that may occur on the test page are considered in advance and written into the script, the script can complete the testing process relatively stably. Conversely, if the current page experiences page changes or abnormal states not covered by the script, subsequent operations may not execute correctly, or even automated script exceptions may occur.

[0062] Reference Figure 3 In related technologies, automated scripts typically employ page text recognition or page element location methods when performing conditional judgments. One approach is to use optical character recognition (OCR) technology to identify text content within page images, such as determining the presence of a target text or keyword (e.g., identifying the three characters for "element A") to confirm if the page is in the expected state. Another approach involves using page structure information provided by the automated testing platform to query controls, element identifiers, element levels, or control attributes. For example, determining if a button or control with a specified identifier exists (e.g., determining if a close button for element C exists). The methods for obtaining page structure may differ across operating systems or testing platforms; page element information can be obtained through their respective automated testing frameworks. Using these methods, the script can locate target elements on the page and execute corresponding operations.

[0063] However, the automation logic in the aforementioned technologies is mainly pre-written by script developers, and the page state judgment and exception handling methods also need to be configured in advance. Therefore, when the position of page controls, displayed text, element identifiers, pop-up styles, or page structure change, the original conditional judgment logic may not be able to accurately identify the current page state, thus causing the preset operations to fail to execute correctly, and its adaptability to page changes not covered during the script recording or writing stage is limited.

[0064] To address the aforementioned problems, this application provides an automated testing method. The automated testing method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0065] Reference Figure 4 , Figure 4 This is a flowchart illustrating an automated testing method provided in an embodiment of this application, such as... Figure 4 As shown in the embodiment of this application, an automated testing method may include steps S410 to S450, which are described in detail below.

[0066] S410. Obtain the visual features of the test page.

[0067] Before each step of the test script is executed, the content of the currently displayed test page is first obtained. The test page refers to the page that the automated test is currently on or displaying, such as the application homepage, a business function page, a page obscured by a pop-up, or a transition page containing a floating layer. The content of this test page is then converted into feature values ​​that characterize the overall layout and main content of the current page, i.e., page visual features. The converted page visual features are used to characterize the overall layout information of the test page, such as the position of pop-ups, the position of the navigation bar, the main content area, and the overall distribution of specific functional areas.

[0068] S420, Obtain the pre-recorded test script.

[0069] The test script in this embodiment is generated during the test preparation phase using a Visual Language Model (VLM). The pre-recorded test script not only records the sequence of test operations but also includes preset visual features and a first operation corresponding to those features. The first operation refers to a pre-recorded operation performed on the test page scene corresponding to the preset visual features, such as clicking a button at a specific coordinate. The preset visual features are visual feature values ​​obtained by feature extraction from the test page during recording. Specifically, these features can be obtained by saving the page image of the test page before executing the first operation and extracting features from that image.

[0070] In the test preparation phase of this application, the visual language model first completes a real operation process and records the operations that need to be repeated during this phase. Simultaneously, it saves the visual features of the corresponding page before each operation, thus forming a correspondence between the page visual features and the first operation, which is then written into the test script. For example, the test script records an entry: when the page visual feature is the hash value of the homepage without pop-ups, the corresponding first operation is clicking the login button at coordinates (100, 200). Therefore, it can be understood that the test script obtained in this application essentially obtains the scenario references and operation references needed for subsequent automated test execution. For example, one item in the test script can correspond to: when the page is in a homepage scenario, execute the first operation of clicking an entry point; another item can correspond to: when the page is in a details page scenario, execute the first operation of swiping upwards. In this way, the test script no longer only includes instructions on what operation should be performed next, but also instructions on what operation should be performed in what page scenario.

[0071] S430. Match the page's visual features with preset visual features and determine the matching result.

[0072] By utilizing the visual features of the current test page, a match is made with the preset visual features associated with the current step to be executed in the test script to determine whether the current test page is compatible with the page scene when the first operation was recorded. The matching result here is used to characterize whether the current test page is consistent with the page scene when the test script was pre-recorded or whether it has reached an executable level. It is understood that in related technologies, recording and playback are executed sequentially according to the order of test operations. However, in this embodiment, a page scene check is performed before each test operation step to avoid continuing to execute the previously recorded operation even if the page state has changed. For example, the current step recorded in the test script should have involved clicking an entry point on an unobstructed homepage. However, if the current page changes its overall scene due to a temporary pop-up window, the matching process in step S430 can identify that the current test page is no longer compatible with the preset visual features in the test script, thus preventing the incorrect execution of the first operation.

[0073] S440. If the matching result meets the preset matching conditions, execute the first operation in the test script corresponding to the preset visual features.

[0074] If the matching result meets the preset matching conditions (e.g., a successful match), it means that the current test page is compatible with the page scenario recorded when the test operation step was recorded. The current test page and the page scenario recorded in the test script have reached the level where the pre-recorded first operation can be executed. Therefore, the automated test operation can be directly executed according to the first operation content pre-recorded in the test script. Since the embodiments of this application have completed the verification between the current test page and the page scenario recorded in the test script before executing the first operation, it can be considered that the current test page is still suitable for execution using the pre-recording method if the preset matching conditions are met, without having to call the visual language model again to participate in the processing, thereby reducing the number of calls to the visual language model and the cost of automated test execution. For example, during the test preparation phase, the visual language model pre-records the action of clicking the search box after entering the homepage, and records the preset visual features of the homepage scene corresponding to this action into the test script. Then, when the test is repeated in the future, as long as the current page matches the preset visual features of the homepage scene, the first action of clicking the search box can be executed directly without having to ask the visual language model again to determine whether the search box should be clicked. This allows for the priority reuse of pre-recorded actions when the scene is consistent, improving the execution efficiency of automated testing.

[0075] S450. If the matching result does not meet the preset matching conditions, call the visual language model to process the test page, and execute the second operation based on the processing result of the visual language model.

[0076] If the matching result does not meet the preset matching conditions, it indicates that the current test page may have undergone page changes not covered by the script, such as pop-ups, ad overlays, prompts, abnormal page jumps, or changes in page layout. Therefore, if the current test page does not match the preset visual features recorded in the test script, the originally recorded first operation will not be used. Instead, the visual language model will be called to re-understand the content of the current test page and output a processing result adapted to the current test page state (e.g., "click the close button in the upper right corner of the pop-up"). Then, the corresponding second operation will be executed based on this processing result. The second operation here refers to the operation obtained by re-identifying and judging the current test page based on the visual language model. For example, the test script originally recorded clicking the function entry directly on the homepage, but a new activity pop-up appears at the top of the current test page after entering the homepage, causing the homepage scenario to no longer meet the conditions for executing the first operation. If the original operation of clicking the function entry is replayed at this time, it is easy to click the wrong position or cause the test operation to fail. Therefore, in the case where the matching result does not meet the preset matching conditions, this application embodiment calls the visual language model to take over the processing, re-identifies the current test page, determines whether to close the pop-up window first, return to a certain page first, or perform other adjustment operations first, and then performs the second operation to restore the page to a state where the test script can continue to be executed, thereby ensuring that the automated testing process can continue to be executed under the condition of page changes or abnormal interference, and ensuring the stability of the automated testing process.

[0077] It should be noted that this embodiment does not call the visual language model for page recognition processing in every step of the automated testing. Instead, the visual language model is only invoked when the page's visual features do not meet the preset matching conditions. This is because many test steps in automated testing scenarios are often repeated. If the visual language model were to re-identify the page and generate operations for each step, it would lead to excessive model calls, increasing test execution costs and time. For test steps where the page's visual features match the preset visual features, it indicates that the current test page is still compatible with the page scenario recorded for that step. The first operation recorded and saved during the test preparation phase can be directly reused without further model inference. Therefore, this embodiment first uses page visual feature matching to determine whether the current page is suitable for pre-recorded operations. Only when matching fails is the visual language model invoked to re-understand and process the current page. This allows the visual language model to primarily handle situations where the pre-recorded script cannot directly adapt, such as page changes, pop-up occlusion, and layout anomalies. By employing this approach, the embodiments of this application retain the visual language model's ability to handle abnormal page scenarios while reusing test scripts to reduce model invocation costs, thereby achieving a certain balance between automated test execution efficiency, operating costs, and fault tolerance stability.

[0078] By employing the above technical solution, this application records not only the first operation but also the preset visual features corresponding to the first operation in the test script. During test execution, the visual features of the test page are acquired, and the current test page's visual features are matched with the preset visual features recorded in the test script to determine whether the current test page is still suitable for executing the first operation. This avoids mechanically applying recorded operations when the test page state changes, reducing the problem of automated test execution failures caused by mismatches between recorded operations and actual page states, and improving the stability of automated test execution. If the matching result meets the preset matching conditions, the first operation corresponding to the preset visual features in the test script is directly reused, reducing the number of times the visual language model is called in repetitive test processes, thereby reducing the operating cost of automated testing and improving test execution efficiency. If the matching result does not meet the preset matching conditions, that is, when the current test page is no longer suitable for the recorded first operation, the visual language model is called to process the test page, and the second operation is executed based on the processing result. Therefore, this application reduces the cost of repeated calls to the video language model while retaining the visual language model's ability to handle page changes, pop-up interference, and other situations, thus balancing the cost of automated testing with fault tolerance and improving the stability and adaptability of the entire automated testing method.

[0079] In one feasible implementation, step S410, obtaining the page visual features of the test page, includes: acquiring a page image of the test page; performing a discrete cosine transform on the page image using a perceptual hash algorithm to extract the low-frequency components of the page image; and encoding the low-frequency components into hash values ​​as the page visual features of the test page.

[0080] Capturing the page image of the current test page can be achieved by calling the screenshot interface or image capture function provided by the operating system to take a real-time screenshot of the application page under test. In practical applications, the page image can include the layout structure, control positions, text information, and any pop-ups, advertising overlays, etc., that may appear on the test page. For example, when the test step enters the homepage of an application, the screenshot interface or image capture function can be called to capture the entire display area of ​​the homepage, obtaining a bitmap image including the navigation bar, content area, and bottom menu. Then, the perceptual hashing algorithm pHash is used to perform a Discrete Cosine Transform (DCT) on the captured page image, transforming the image from the spatial domain to the frequency domain. It should be noted that in the frequency domain, the low-frequency components of the image are used to characterize the main layout structure and overall brightness changes, while the high-frequency components correspond to the local details and noise of the image. Since the low-frequency components mainly retain the overall information of the image, they are more suitable for comparing whether two page images belong to the same page scene. For example, there is a scrolling text in the lower right corner of a test page. The text content of the scrolling text changes during two data collections. However, the differences in the text content of these local areas usually mainly affect the high-frequency components, and do not change the overall layout and main structure of the page image. Therefore, such local changes usually do not significantly affect the low-frequency components. The information extracted based on the low-frequency components can reflect the main layout of the page relatively stably.

[0081] The extracted low-frequency components can be encoded into a hash value, which is then used as the visual feature of the test page. In this embodiment, the encoding process typically compares each value in the low-frequency component matrix with a reference value (e.g., the median or average of all values). Bits greater than the reference value are set to 1, and bits less than or equal to the reference value are set to 0, ultimately resulting in a fixed-length binary hash value. Following the example above, after binarizing the low-frequency components of the application's homepage, a 64-bit binary number can be obtained, such as "10110011...". This hash value represents the overall scene characteristics of the application's homepage. In subsequent matching, the automated testing tool only needs to compare the difference between the current page's hash value and a preset hash value in the script to quickly determine whether the two pages belong to the same scene. Because the hash value has a fixed length and fast calculation speed, it can also adapt to the high-frequency matching requirements in automated testing, ensuring the stability of the automated testing process.

[0082] In one feasible implementation, step S430, which involves matching the page visual features with preset visual features to determine the matching result, includes: determining the feature difference value between the page visual features and the preset visual features; comparing the feature difference value with a preset difference threshold; and determining the matching result based on the comparison result.

[0083] To improve the applicability of page scene matching processing and determine the feature difference value between page visual features and preset visual features, the difference can be calculated using a corresponding difference measurement method based on the specific representation of the page visual features. When page visual features are represented using encoded sequences, the feature difference value can be determined based on the number of differences between the encoded sequences; when page visual features are represented using feature vectors, the feature difference value can be determined based on the distance between the feature vectors; when page visual features are represented using a combination of multiple parts, the local difference values ​​between each part can be determined first, and then the overall feature difference value can be determined based on these local difference values.

[0084] After determining the feature difference value, it is compared with a preset difference threshold. The preset difference threshold is a pre-defined numerical limit, and its specific value can be configured according to the tolerance of the test scenario for page changes. When the feature difference value is within the range allowed by the preset difference threshold, the current test page is considered to have sufficient similarity to the page corresponding to the recorded test script, thus determining that the matching result meets the preset matching conditions. When the feature difference value exceeds the range allowed by the preset difference threshold, it is considered that there is a significant difference between the current test page and the page scene at the time of recording, thus determining that the matching result does not meet the preset matching conditions.

[0085] In one feasible implementation, determining the feature difference value between the page visual feature and the preset visual feature includes: obtaining the first hash value corresponding to the page visual feature and the second hash value corresponding to the preset visual feature; comparing the first hash value and the second hash value bit by bit; counting the number of bits that are different between the first hash value and the second hash value, and determining them as the feature difference value between the page visual feature and the preset visual feature.

[0086] As mentioned earlier, one encoding method for page visual features is to extract and encode binary hash values ​​from the test page image using a perceptual hash algorithm. Similarly, preset visual features are also binary hash values ​​extracted and stored from an image of a specific page scene during the recording phase. It can be understood that both the page visual features and preset visual features have the same bit length (e.g., 64 bits or 128 bits) and are used to characterize the low-frequency component information of their respective pages. To facilitate subsequent calculations, the automated testing tool reads the first hash value of the current test page and the second hash value of the preset visual feature corresponding to the current test step from memory or storage media. For example, the first hash value is the binary string 1011001110001110, and the second hash value is 1011001110011110.

[0087] Next, the first hash value and the second hash value are compared bit by bit. The values ​​at the same positions (bits) in the two binary strings are compared one by one to determine if the two values ​​at that position are the same. Specifically, it can be done by iterating from the 1st bit to the Nth bit (N is the total number of bits in the hash value). For each position, it is checked whether the value of the first hash value (0 or 1) at that bit is equal to the value of the second hash value at that bit. In practical implementation, a bitwise XOR operation can also be used instead of bit-by-bit comparison: the first hash value and the second hash value are XORed to obtain a new binary string. A bit in the string being 1 indicates that the two hash values ​​at that position are different, and a bit being 0 indicates that they are the same. For example, performing a bitwise XOR operation on the first hash value 1011001110001110 and the second hash value 1011001110011110 results in 0000000000010000, where only one bit is 1 and the rest are 0. Next, count the number of bits that differ between the first and second hash values, i.e., count the number of 1s in the XOR result. In the XOR result 0000000000010000, there is only one 1, therefore the number of different bits is 1. This value is the feature difference value. The smaller the feature difference value, the more similar the visual features of the two pages are; the larger the feature difference value, the greater the difference in the overall low-frequency structure of the two pages. For example, if the feature difference value is 0, it means that the two hash values ​​are exactly the same, and the page scene is completely identical; if the feature difference value is large (e.g., exceeding 10), it means that there is a significant difference between the current page and the page scene at the time of recording, and the current page may have new pop-ups or layout changes. By comparing each bit and counting the different bits, the visual similarity between two pages can be quickly and efficiently quantified, ensuring the stable execution of the automated testing process.

[0088] In one feasible implementation, the preset difference threshold is determined based on the page change frequency of the test page. Page change frequency refers to how frequently the displayed content on the test page changes during multiple automated test executions. For pages with stable layouts and no dynamic content, such as settings pages and form filling pages of the application under test, the page change frequency is low; while for homepage pages containing ad carousels, information feed refreshes, and dynamic recommendations, the page change frequency is high. Therefore, a preset frequency threshold can be set to distinguish whether a page tends to be static or dynamic. In practical use, the page change frequency can be determined by statistically analyzing the average change interval of the test page over multiple historical executions or by manual annotation.

[0089] Specifically, when the page change frequency is less than a preset frequency threshold, the first difference threshold is determined as the preset difference threshold; when the page change frequency is greater than or equal to the preset frequency threshold, the second difference threshold is determined as the preset difference threshold; and the second difference threshold is greater than the first difference threshold.

[0090] When the page change frequency is less than the preset frequency threshold, it indicates that the page content is relatively fixed and the accuracy requirement for visual matching is high. In this case, the first difference threshold can be set as the preset difference threshold. The first difference threshold is a threshold applicable to low-change-frequency page scenarios. Its value is smaller than the second difference threshold, meaning that only a small range of visual differences are allowed between the current test page and the pre-recorded page scenario. For example, if the page visual features are encoded using hash value encoding, the first difference threshold can be set to 5, meaning that only when the two pages are highly consistent in low-frequency visual structure (e.g., the number of bits that differ in the hash values ​​of the two pages is less than 5) will a match be considered successful, thus avoiding pages with large differences from being misjudged as matches. Conversely, when the page change frequency is greater than or equal to the preset frequency threshold, it indicates that there are dynamic change areas on the page that are permissible and do not affect core operations (such as corner ad image switching or recommended content updates). In this case, the second difference threshold should be set as the preset difference threshold. The second difference threshold can be set to a value greater than the first difference threshold. For example, if the page visual features are encoded using hash value encoding, the second difference threshold can be set to 15. This ensures that even if the page changes to a certain extent, as long as the overall layout and the positions of the main controls do not change significantly (e.g., the number of bits that differ between the hash values ​​of the two pages is less than 15), a successful match can still be determined, thereby avoiding frequent VLM calls due to normal changes in non-core areas.

[0091] In one feasible implementation, the reusable first operation recorded in the test script is not limited to a single type of page interaction action. Instead, it can be recorded as multiple operation forms adapted to different page interaction needs, based on the actual test process during the recording phase. Specifically, the first operation includes at least one of click, swipe, and input operations. Here, a click operation refers to the operation form that performs a trigger action on a target position on the test page; a swipe operation refers to the operation form that moves the displayed content or switches page areas along a predetermined direction on the test page; and an input operation refers to the operation form that writes characters, numbers, or other information to the input area on the test page.

[0092] Furthermore, step S440, which states that if the matching result meets the preset matching conditions, the first operation corresponding to the preset visual feature in the test script is executed, includes: if the matching result meets the preset matching conditions, obtaining the operation coordinates and operation type corresponding to the preset visual feature; and executing the first operation on the test page according to the operation coordinates and operation type.

[0093] Because the test script pre-stores the correspondence between preset visual features and the first operation, and the first operation pre-recorded during the test preparation phase includes operation coordinates, operation type, and operation parameters corresponding to the operation type, when the matching result meets the preset matching conditions (i.e., the current page visual feature successfully matches a preset visual feature in the script), the operation type (e.g., click, swipe, or input) corresponding to the preset visual feature and the operation parameters associated with that operation type are read from the mapping relationship. For a click operation, the operation parameters may specifically include the click coordinates (e.g., x-coordinate and y-coordinate); for a swipe operation, the operation parameters may specifically include the coordinates of the swipe start point and end point; for an input operation, the operation parameters may specifically include the coordinates of the input box position and the text content to be entered. For example, after a successful match, the operation type can be obtained from the script as click, and the operation coordinates are (350, 800), and these operation coordinates were generated and recorded by the visual language model based on the page scene at that time during the recording phase.

[0094] Since the current page's visual features successfully match the preset visual features during recording, it indicates a high degree of consistency in the overall low-frequency structure of the two pages. The operation coordinates determined during recording remain valid on the current test page. Therefore, based on the obtained operation coordinates and operation type, the first operation can be executed on the test page. Using the example above, the first operation can be performed at coordinates (350, 800) on the test page. Of course, when actually executing the first operation, the corresponding automated execution interface can be called based on the obtained operation type: if the operation type is a click, a finger or mouse is simulated to click at the operation coordinates; if it is a swipe, the trajectory from the starting point coordinates to the ending point coordinates is simulated; if it is input, the input box position coordinates are clicked first to activate the input focus, and then the text content carried in the operation parameters is entered. This avoids calling the visual language model for each operation, allowing for quick completion of automated operations using pre-recorded operation coordinates, and also reduces the execution cost of automated testing.

[0095] It is understood that the page visual features in this application embodiment are mainly used to represent the overall layout of the page, such as the general layout of the page, the location of the main areas, and the location of the core function entry points. Therefore, local differences such as changes in local text, local images, and scrolling content on the page usually do not directly affect the judgment of the overall page layout. In other words, as long as the overall layout and main operation areas of the current page are basically consistent with those during recording, even if there are a few changes in local content on the page, it can be considered that the current page is still suitable for performing the first operation saved during the recording stage.

[0096] In practical applications, high-frequency automated testing typically targets core business pages or frequently used pages in an application, such as the homepage, search page, order page, and release page. These pages are usually frequently used by users, and the entry points and buttons that users frequently click are generally designed to be prominent, and the clickable areas are relatively easy to trigger. Therefore, the page layout usually does not frequently undergo small shifts. If the page visual feature matching passes, it means that the overall layout of the current page is basically consistent with the page during recording. At this time, the first operation can be executed directly according to the operation coordinates and operation type saved during the recording stage, and the page interaction can be completed normally, thus ensuring the stable reuse of the test script. However, if the test page does undergo layout changes that affect the accuracy of the operation, such as the position of the main function button that the user needs to click being moved to a significantly different position from that during recording, the area of ​​the page where the user mainly operates being adjusted, or the page being obscured by a pop-up window, then the overall layout of the page will usually change accordingly, leading to the failure of page visual feature matching. In this case, a visual language model can be used to process the current test page, thereby ensuring the stable execution of automated test operations.

[0097] In one feasible implementation, step S450, which involves calling a visual language model to process the test page if the matching result does not meet the preset matching conditions, and performing a second operation based on the processing result of the visual language model, includes: if the matching result does not meet the preset matching conditions, constructing preset prompt information; wherein the preset prompt information includes at least an operation instruction description and a desired output format, the operation instruction description instructing the visual language model to identify the page elements to be processed in the test page and output target page operation information, and the desired output format constraining the visual language model to return structured target page operation information; the target page operation information includes element position coordinates and target page operation type; inputting the acquired test page image and the preset prompt information together into the visual language model; receiving the target page operation information output by the visual language model and parsing the target page operation information to obtain element position coordinates and target page operation type; and performing a second operation in the test page based on the element position coordinates and target page operation type.

[0098] If the matching result does not meet the preset matching conditions, it indicates that an anomaly or change has occurred on the current page that was not recorded by the script (e.g., an unknown pop-up, an advertising overlay, or a change in the overall layout). In this case, the pre-recorded operation is no longer used directly, but the Visual Language Model (VLM) is used for processing. To facilitate the VLM's understanding of the current automated test's processing intent, this embodiment of the application can construct a preset prompt message before inputting the page image of the current test page into the VLM. The preset prompt message includes at least two parts: an operation instruction description and a desired output format. The operation instruction description can be an instruction written in natural language, used to instruct the VLM to identify the page elements to be processed on the test page (e.g., pop-ups, close buttons, confirmation boxes, etc.) and output the corresponding target page operation information. The desired output format is used to constrain the VLM to return results in the form of structured data. Specifically, it can be specified that the returned JSON or XML structure must include the element position coordinates (e.g., x and y values) and the target page operation type (e.g., click, close, confirm, etc.). In this way, the preset prompt message enables the automated testing tool to stably parse the processing results returned by the VLM. For example, in a specific test scenario, the constructed preset prompt message can be: "Please identify the pop-up window on the current page, find its close button, and output the following format: {'operation':'click', 'coordinates':(x,y)}".

[0099] Then, the screenshot interface is used to obtain the image of the current test page. This image, along with the constructed preset prompt information, is sent as input to the visual language model. Based on the input image and the preset prompt information, the visual language model infers the semantic meaning of each element in the image and outputs the corresponding operation information according to the format required by the preset prompt information. Continuing with the example above, after inputting a screenshot of the current page containing the pop-up and the preset prompt information "Please identify the pop-up in the current page, find its close button, and output the format: {'operation':'click', 'coordinates': (x, y)}" into the visual language model, the model can analyze the screenshot using visual understanding, locate the close button on the pop-up (e.g., the "×" icon in the upper right corner), determine the button's coordinates in the image, and finally output a processing result that conforms to the expected output format in the preset prompt information, such as "{'operation':'click', 'coordinates': (340, 120)}". After obtaining the target page operation information output by the visual language model, this information is parsed to obtain the element position coordinates and the target page operation type. Since the structured form of the returned processing result has been constrained by the expected output format in the preceding steps, a corresponding parser (such as a JSON parser) can be used to convert the text output by the visual language model into a programmable data structure, from which the element position coordinates and the target page operation type can be extracted. For example, for the VLM-returned {'operation': 'click', 'coordinates': (340, 120)}, after parsing, the target page operation type is "click" and the element position coordinates are (340, 120). These two pieces of information constitute the operation parameters required to execute the second operation. Then, the corresponding automated execution interface (such as click, swipe, or input interface) is called to execute the operation at the specified coordinates on the test page. Taking the above parsing result as an example, the automated testing tool can execute a click operation at coordinates (340, 120) on the test page to close the pop-up window that is obscuring the homepage content. Therefore, this embodiment of the application can utilize the visual understanding capability of the visual language model to handle page elements not anticipated during the test preparation phase when matching fails, ensuring that the automated testing process can continue to execute, while reducing the need for manual intervention or modification of test scripts. It should be noted that clicking to close the pop-up in the above example is only one case of the second operation. In actual applications, VLM may also return other operation types (such as swiping, input, etc.), and their execution principle is similar to the above process, which will not be repeated here.

[0100] In one feasible implementation, after performing the second operation based on the processing result of the visual language model, the method of this application embodiment further includes: recording the operation coordinates, operation type, and operation order corresponding to the second operation; establishing a correspondence between the page visual features of the test page and the second operation; and updating the test script based on the page visual features and the second operation corresponding to the page visual features.

[0101] To ensure that the operations determined by the visual language model can be reused in subsequent automated testing, the operation coordinates, operation type, and operation order can be recorded after the second operation is executed. The operation coordinates can be the actual location of the second operation on the test page, such as the coordinates of a click, the start and end of a swipe, or the location of an input box. The operation type can be a page interaction type such as click, swipe, or input. The operation order indicates the execution sequence of the second operation within the current test flow (e.g., which step in the entire test flow it is, or its position relative to the previously recorded operation). For example, assuming the current test script already contains the first 5 steps, and the visual language model executes a click to close a pop-up window, the automated testing tool can record the operation type as a click, the operation coordinates as (340, 120), and the operation order as step 6.

[0102] Furthermore, automated testing tools can establish a correspondence between the visual features of the current test page and the recorded second operation. As mentioned earlier, before calling the visual language model, the page image of the current test page has been captured and its visual features (i.e., the visual features of the abnormal page that caused the matching failure) have been extracted. The captured page visual features represent the specific page scenario that triggered the intervention of the visual language model. Therefore, a mapping relationship can be established between this page visual feature and the second operation that was just executed and recorded, similar to the mapping relationship between the preset visual features and the first operation established during the recording process in the test preparation phase. In this way, when a page scenario with the same or similar page visual features appears again in the subsequent automated testing process, the automated testing tool can directly obtain the corresponding operation content based on this correspondence, without having to call the visual language model again for judgment each time. Furthermore, the page visual feature can also be written into the test script as a new preset visual feature, and the second operation can be written into the corresponding position in the test script as the operation corresponding to the page visual feature, or written into the test script as a supplementary processing branch of the current test step. The updated test script includes not only the correspondence between the preset visual features obtained during the recording phase and the first operation, but also the correspondence between the page visual features supplemented by the visual language model during actual execution and the second operation. Therefore, when encountering the same or similar page change scenarios again, the recorded operations can be executed first through the correspondence in the test script, thereby reducing repeated calls to the visual language model and improving the test script's coverage of page change scenarios.

[0103] In a feasible specific application embodiment, the automated testing method provided in this application can be applied to the performance automation testing scenario of client applications. Taking the repeated testing of the homepage entry, function entry click, and page jump process of a client application as an example, a test preparation phase can be entered before the formal execution of the automated test. In the test preparation phase, the automated testing tool calls the visual language model to identify the page displayed on the current device, and the visual language model determines the page operation to be performed at each step according to the test objective. For example, when an activity pop-up, permission prompt box, or advertising overlay appears on the page, the visual language model can first identify the pop-up or overlay and output the operation of closing the pop-up or overlay; after the page returns to the expected test step flow, the visual language model continues to identify the function entry on the page and outputs the corresponding click, swipe, or input operation. Through the above test preparation process, the visual language model can complete the automated process to be repeated before the formal repeated execution of the test, and process the page elements and temporary pop-ups that may appear in the automated process.

[0104] During the initial automated operation using a visual language model, the automated testing tool simultaneously records each step of the automated process and saves the visual features of the current test page before each step is executed. In other words, the automated testing tool doesn't just record the actions themselves, such as clicking a coordinate or swiping an area, but also records the page context preceding the action, establishing a correspondence between page visual features and action steps. For example, before clicking a function entry on the homepage, the automated testing tool first captures an image of the homepage and generates corresponding page visual features, then records the correspondence between these visual features and the coordinates and type of the action taken to click the function entry. This way, the recorded test script indicates the specific page context in which an action should be performed, rather than simply the order in which actions are executed.

[0105] After completing the recording in the preparation phase, the formal automated test execution phase can begin. The automated testing tool executes each test step sequentially according to the recorded test script. Before executing each step in the test script, the content of the test page currently displayed on the device is captured, and the page's visual features are generated based on the test page content. Then, the page's visual features are matched with the preset visual features pre-recorded in the test script for that step. If the matching result shows that the current page matches the page scene when the step was recorded, it means that the current page is still suitable for executing the pre-recorded operation corresponding to that test step. In this case, the automated testing tool can directly execute the automated operation recorded in the test script, such as clicking coordinates saved during recording, sliding along the recorded trajectory, or writing text content at a specified input location. After execution, the automated testing tool continues to the next step in the test script and repeats the above process of page visual feature matching and operation execution.

[0106] If, before a certain test step is executed, the visual features of the current page do not match the preset visual features recorded in the test script, it indicates that the current page is inconsistent with the page scene recorded during the recording of this step. This may be due to pop-ups, overlays, page state changes, or other interface interference not covered during the recording phase. In this case, the automated testing tool will not continue to directly execute the pre-recorded operations in this step, but will instead call the visual language model again to identify and process the current page. The visual language model can determine the appropriate processing action based on the current page image and the test target, such as closing the current pop-up, returning to the target page, clicking the confirmation button, or performing other operations to restore the page to a state where the test script can continue to execute. After this processing action is executed, the automated testing tool can continue to execute subsequent steps according to the test script. Therefore, this embodiment of the application can prioritize the reuse of recorded scripts during repeated testing to reduce model call costs, and uses the visual language model to handle abnormal pages when the page scene does not match, thereby improving the execution stability of the automated testing process.

[0107] The above describes an automated testing method provided by an embodiment of this application. The following will describe the apparatus for performing the above-described automated testing method. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of an automated testing device provided in an embodiment of this application. Figure 5 As shown, the automated testing device 500 includes:

[0108] The feature acquisition unit 501 is used to acquire the visual features of the test page;

[0109] The script acquisition unit 502 is used to acquire a pre-recorded test script; wherein the test script includes preset visual features and a first operation corresponding to the preset visual features;

[0110] The matching determination unit 503 is used to match the page visual features with the preset visual features and determine the matching result;

[0111] The first execution unit 504 is used to execute a first operation corresponding to the preset visual feature if the matching result meets the preset matching conditions.

[0112] The second execution unit 505 is used to call the visual language model to process the test page if the matching result does not meet the preset matching conditions, and to perform a second operation based on the processing result of the visual language model.

[0113] Optionally, the feature acquisition unit 501 is specifically used to: acquire a page image of the test page; perform discrete cosine transform on the page image using a perceptual hash algorithm to extract the low-frequency components of the page image; and encode the low-frequency components into hash values ​​as page visual features of the test page.

[0114] Optionally, the matching determination unit 503 is specifically used to: determine the feature difference value between the page visual features and the preset visual features; compare the feature difference value with the preset difference threshold, and determine the matching result based on the comparison result.

[0115] Optionally, the matching determination unit 503 is specifically configured to: obtain a first hash value corresponding to the page visual feature and a second hash value corresponding to the preset visual feature; compare the first hash value and the second hash value bit by bit; count the number of bits that differ in value between the first hash value and the second hash value, and determine the feature difference value between the page visual feature and the preset visual feature. Optionally, the preset difference threshold is determined based on the page change frequency of the test page; wherein, when the page change frequency is less than a preset frequency threshold, the first difference threshold is determined as the preset difference threshold; when the page change frequency is greater than or equal to the preset frequency threshold, the second difference threshold is determined as the preset difference threshold; the second difference threshold is greater than the first difference threshold.

[0116] Optionally, the first operation includes at least one of a click operation, a swipe operation, and an input operation; the first execution unit 504 is specifically used for: if the matching result satisfies the preset matching condition, obtaining the operation coordinates and operation type corresponding to the preset visual feature; and executing the first operation on the test page according to the operation coordinates and the operation type.

[0117] Optionally, the second execution unit 505 is specifically configured to: if the matching result does not meet the preset matching conditions, construct preset prompt information; wherein the preset prompt information includes at least an operation instruction description and a desired output format, the operation instruction description is used to instruct the visual language model to identify the page elements to be processed in the test page and output target page operation information, and the desired output format is used to constrain the visual language model to return structured target page operation information; the target page operation information includes element position coordinates and target page operation type; input the acquired page image of the test page and the preset prompt information together into the visual language model; receive the target page operation information output by the visual language model, and parse the target page operation information to obtain element position coordinates and target page operation type; and perform the second operation in the test page according to the element position coordinates and the target page operation type.

[0118] Optionally, the device further includes: a script update unit; the script update unit is specifically used to: record the operation coordinates, operation type, and operation sequence corresponding to the second operation; establish a correspondence between the page visual features of the test page and the second operation; and update the test script based on the page visual features and the second operation corresponding to the page visual features.

[0119] This application also provides an electronic device in its embodiments. (See reference...) Figure 6 The diagram illustrates a structural schematic of an electronic device suitable for implementing the automated testing method in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, personal digital assistants (PDAs), tablet computers (PADs), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0120] like Figure 6As shown, the electronic device may include a processing unit (e.g., a central processing unit (CPU), graphics processing unit (GPU), etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or loaded from storage device 608 into random access memory (RAM) 603. When the electronic device is powered on, RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0121] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0122] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the automated testing methods provided in this application.

[0123] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the automated testing methods provided in this application.

[0124] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause an electronic device (which may be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in the various embodiments of this application.

[0126] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0127] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An automated testing method, characterized in that, include: Obtain the visual features of the test page; Obtain a pre-recorded test script; wherein the test script includes preset visual features and a first operation corresponding to the preset visual features; The page visual features are matched with the preset visual features to determine the matching result; If the matching result meets the preset matching conditions, execute the first operation in the test script corresponding to the preset visual feature; If the matching result does not meet the preset matching conditions, the visual language model is invoked to process the test page, and a second operation is performed based on the processing result of the visual language model.

2. The method according to claim 1, characterized in that, The acquisition of the visual features of the test page includes: Capture the page image of the test page; The low-frequency components of the page image are extracted by performing a discrete cosine transform on the page image using a perceptual hash algorithm. The low-frequency components are encoded into hash values, which are used as the visual features of the test page.

3. The method according to claim 1, characterized in that, The step of matching the page visual features with the preset visual features to determine the matching result includes: Determine the feature difference value between the page visual features and the preset visual features; The feature difference value is compared with a preset difference threshold, and the matching result is determined based on the comparison result.

4. The method according to claim 3, characterized in that, Determining the feature difference value between the page visual features and the preset visual features includes: Obtain the first hash value corresponding to the page visual feature and the second hash value corresponding to the preset visual feature; Compare the first hash value and the second hash value bit by bit; The number of bits that differ in value between the first hash value and the second hash value is counted and determined as the feature difference value between the page visual feature and the preset visual feature.

5. The method according to claim 3, characterized in that, The preset difference threshold is determined based on the page change frequency of the test page; wherein, when the page change frequency is less than a preset frequency threshold, a first difference threshold is determined as the preset difference threshold; when the page change frequency is greater than or equal to the preset frequency threshold, a second difference threshold is determined as the preset difference threshold; the second difference threshold is greater than the first difference threshold.

6. The method according to claim 1, characterized in that, The first operation includes at least one of a click operation, a swipe operation, and an input operation; if the matching result meets the preset matching conditions, the first operation corresponding to the preset visual feature in the test script is executed, including: If the matching result meets the preset matching condition, obtain the operation coordinates and operation type corresponding to the preset visual feature; The first operation is performed on the test page according to the operation coordinates and the operation type.

7. The method according to claim 1, characterized in that, If the matching result does not meet the preset matching conditions, the visual language model is invoked to process the test page, and a second operation is performed based on the processing result of the visual language model, including: If the matching result does not meet the preset matching conditions, a preset prompt message is constructed; wherein, the preset prompt message includes at least an operation instruction description and a desired output format, the operation instruction description is used to instruct the visual language model to identify the page elements to be processed in the test page and output the target page operation information, and the desired output format is used to constrain the visual language model to return structured target page operation information; the target page operation information includes element position coordinates and target page operation type; The collected page image of the test page and the preset prompt information are input into the visual language model; Receive the target page operation information output by the visual language model, and parse the target page operation information to obtain the element position coordinates and the target page operation type; Based on the element's position coordinates and the target page's operation type, the second operation is performed on the test page.

8. The method according to claim 1, characterized in that, After performing the second operation based on the processing result of the visual language model, the method further includes: Record the operation coordinates, operation type, and operation sequence corresponding to the second operation; Establish the correspondence between the visual features of the test page and the second operation; The test script is updated based on the page's visual features and the second operation corresponding to those features.

9. An automated testing device, characterized in that, include: The feature acquisition unit is used to acquire the visual features of the test page; A script acquisition unit is used to acquire a pre-recorded test script; wherein the test script includes preset visual features and a first operation corresponding to the preset visual features; The matching determination unit is used to match the page visual features with the preset visual features and determine the matching result; The first execution unit is configured to perform a first operation corresponding to the preset visual feature if the matching result meets the preset matching conditions. The second execution unit is used to call the visual language model to process the test page if the matching result does not meet the preset matching conditions, and to perform a second operation based on the processing result of the visual language model.

10. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the automated testing method as described in any one of claims 1 to 8.

11. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the automated testing method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the automated testing method as described in any one of claims 1 to 8.

Images