Web automatic testing method, system and device, medium and program product

Abstract

The invention discloses a Web automatic testing method, system and device, a medium and a program product, and belongs to the technical field of artificial intelligence. The method comprises the steps that test case abstract representation information is generated according to interactive input of a user and a test case mapping template, and the test case abstract representation information comprises a target test task type indicated by the interactive input and intention natural language description information; based on the test case abstract representation information, obtaining a target screenshot of the tested Web page and attribute information of interaction elements; a multi-mode large language model is called to conduct reasoning based on the target screenshot, the attribute information of the interaction elements and the intention natural language description information, reasoning result information is output, and the reasoning result information represents operation executed on the target interaction elements located at the target position in the tested Web page; and converting the reasoning result information into a machine executable instruction, and executing the machine executable instruction. According to the embodiment of the invention, the stability of Web automatic testing can be improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a Web automatic testing method, system, device, medium and program product. BACKGROUND

[0002] Web automatic testing is an important means of verifying Web functions. Testers can write test scripts according to Web pages to be tested, and simulate user operation behaviors in a browser by using an automatic tool to run the written test scripts, so as to test whether the Web pages meet the expected effect. Testers rely on the user interface of the Web page when writing the test scripts, and the user interface of the Web page will change with the update of the version. When there is any change in the user interface of the Web page, such as adding a field, changing the position of a button, or even upgrading a frame, the positioning logic in the original test script is invalid, thereby causing the test script to be invalid, and greatly reducing the stability of Web automatic testing. SUMMARY

[0003] The application embodiment provides a Web automatic testing method, system, device, medium and program product, which can improve the stability of Web automatic testing.

[0004] In a first aspect, the application embodiment provides a Web automatic testing method, comprising: generating test case abstract representation information by an interactive front end according to user interaction input and a preset test case mapping template, the test case abstract representation information including a target test task type indicated by the interaction input and intention natural language description information; obtaining, by a test end, a target screenshot of a Web page to be tested and attribute information of an interactive element in the Web page to be tested based on the test case abstract representation information; calling, by the test end, a multi-modal large language model to perform reasoning based on the target screenshot, the attribute information of the interactive element and the intention natural language description information, and outputting reasoning result information, the reasoning result information representing an operation performed on a target interactive element located at a target position in the Web page to be tested; converting, by the test end, the reasoning result information into machine executable instructions, and executing the machine executable instructions to implement Web automatic testing.

[0005] In a second aspect, the embodiments of the present application provide a Web automatic testing system, comprising: an interactive front-end device configured to generate test case abstract representation information according to user interaction input and a preset test case mapping template, the test case abstract representation information comprising a target test task type indicated by the interaction input and intention natural language description information; a test terminal device communicatively connected to the interactive front-end device and configured to: acquire a target screenshot of a Web page under test and attribute information of an interactive element in the Web page under test based on the test case abstract representation information; call a multi-modal large language model to perform reasoning based on the target screenshot, the attribute information of the interactive element, and the intention natural language description information, and output reasoning result information representing an operation performed on a target interactive element located at a target position in the Web page under test; and convert the reasoning result information into machine executable instructions and execute the machine executable instructions to implement Web automatic testing.

[0006] In a third aspect, the embodiments of the present application provide a Web automatic testing device, comprising: a processor and a memory storing computer program instructions; and the processor implements the Web automatic testing method of the first aspect when executing the computer program instructions.

[0007] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, the computer readable storage medium storing computer program instructions, and the computer program instructions are executed by a processor to implement the Web automatic testing method of the first aspect.

[0008] In a fifth aspect, the embodiments of the present application provide a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the Web automatic testing method of the first aspect.

[0009] The embodiment of the application provides a Web automatic testing method, system, device, medium and program product, which can automatically generate test case abstract representation information capable of representing user test intention according to user interaction input and a preset test case mapping template, obtain target screenshot of a tested Web page and attribute information of an interactive element in the tested Web page based on the test case abstract representation information, and call a multi-modal large language model to perform reasoning based on multi-modal data such as the target screenshot, the attribute information of the interactive element and intention natural language description information, so as to obtain reasoning result information capable of representing an operation performed on a target interactive element located at a target position in the tested Web page, so as to convert the reasoning result information into machine executable instructions and perform the operation, and realize Web automatic testing. The multi-modal large language model can automatically locate the interactive element and the position of the interactive element required by the test task through visual understanding capability, natural language understanding capability and reasoning capability of the multi-modal large language model without the need of the user to define the specific position of the operated interactive element, even if the user interface of the tested Web page is changed, the multi-modal large language model can accurately locate the interactive element and the position of the interactive element required by the test task, without the need of the user to manually modify or recompile the test script, and the stability of the Web automatic testing is improved. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings required to be used in the embodiments of the application will be briefly introduced. Those skilled in the art can obtain other drawings according to these drawings without creating any creative labor.

[0011] FIG. 1 The schematic diagram of the architecture of the Web automatic testing system provided by an embodiment of the application is shown in the figure.

[0012] FIG. 2 The schematic diagram of the architecture of the Web automatic testing system provided by another embodiment of the application is shown in the figure.

[0013] FIG. 3 The flowchart of the Web automatic testing method provided by an embodiment of the application is shown in the figure.

[0014] FIG. 4 The schematic diagram of an example of the interactive interface provided by an embodiment of the application is shown in the figure.

[0015] FIG. 5 The schematic diagram of an example of the test terminal processing flow provided by an embodiment of the application is shown in the figure.

[0016] FIG. 6 The schematic diagram of an example of the target screenshot of the tested Web page provided by an embodiment of the application is shown in the figure.

[0017] FIG. 7for FIG. 6 A schematic diagram illustrating an example of a screenshot of a target element with its interactive element location marked.

[0018] FIG. 8 A schematic diagram of an example of a test terminal provided in an embodiment of this application;

[0019] FIG. 9 This is a schematic diagram of the structure of a Web automation testing device provided in an embodiment of this application. Detailed Implementation

[0020] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples. It should be noted that the acquisition, storage, use, and processing of information and data in the embodiments of this application are all authorized by users or relevant organizations and comply with the relevant provisions of national laws and regulations.

[0021] Web automation testing is an important means of verifying web functions. Testers can write test scripts according to the web pages to be tested, simulate user behaviors in the browser by using the automatic tool to run the written test scripts, and test whether the web pages meet the expected effect. Testers rely on the user interface of the web page when writing test scripts, and the user interface (User Interface, UI) of the web page changes with the update of the version. Even if the function of the web page does not change substantially, when there is any change in the user interface of the web page, such as adding a field, changing the position of the button or even upgrading the frame, the positioning logic in the original test script is invalid, thereby causing the test script to be invalid and greatly reducing the stability of the web automation testing. If the test script is to be effective, testers need to manually modify the test script or even manually rewrite the test script, which reduces the efficiency of web automation testing. The writing of the test script also requires testers to have a high level of use of the program language, and for users who have a poor command of the program language, the difficulty of web automation testing is higher, which increases the cost of web automation testing, excludes users who are rich in business knowledge but lack coding ability from the web automation testing process, and easily causes business islands and business flow bottlenecks in the whole web business process. Moreover, the traditional test-driven framework interacts with the invisible Document Object Model (DOM) through the Application Programming Interface (API), and therefore, the traditional test-driven framework cannot perceive the rendering effect of the displayed web page, and cannot detect visual problems such as element overlap, color error and layout disorder, thereby reducing the effect of web automation testing.

[0022] The application provides a Web automation testing method, system, device, medium and program product, which can convert natural language description information input by a user for describing a test intention into test case abstract representation information, simplifies a test case writing process of the user, and improves test case writing efficiency. A multimodal large language model is introduced to infer multimodal data obtained by fusing images and structured information from a Web page, visual understanding capability of the multimodal large language model is utilized to automatically locate an interactive element on the Web page and perceive a rendering effect of the Web page, visual problems of the Web page can be detected, and Web automation testing efficiency is improved. If a user interface of the Web page is changed due to upgrading or other reasons, the interactive element on the Web page can also be accurately located, machine executable instructions converted from inference result information obtained by inference are interacted with the interactive element, a test script does not need to be manually modified or re-written by the user, Web automation testing can be realized, and Web automation testing efficiency is improved. Even a user who lacks coding ability can realize Web automation testing by inputting natural language description, Web automation testing difficulty is reduced, Web automation testing cost is reduced, business islands and business flow bottleneck in a whole Web business process can be solved, and Web business development efficiency is improved.

[0023] The Web automation testing method, system, device, medium and program product provided by the application are described below.

[0024] For ease of understanding, the application architecture of the Web automation testing method provided by the application, i.e., the structure of the Web automation testing system, is simply described. FIG. 1 The architecture diagram of the Web automation testing system provided by an embodiment of the application is shown in FIG. 1. FIG. 1 As shown in FIG. 1, the Web automation testing system can include an interactive front-end device 11 and a test-end device 12.

[0025] The interactive front-end device 11 can perform human-computer interaction with the user, receive the interactive input of the user, and perform corresponding specific operations in response to the interactive input. In some examples, the interactive front-end device 11 has an input unit, which can include but is not limited to a display panel, a touch panel, a keyboard, a mouse, a joystick, a microphone, and the like. The interactive front-end device 11 can realize human-computer interaction with the user through the input unit, for example, the interactive front-end device 11 can display an interactive interface to the user through the display panel or the touch panel, receive the click, long press, sliding, and the like of the user on the interactive interface through the touch panel, the keyboard, the mouse, the joystick, and the like, and can also receive the voice input of the user through the microphone. The interactive front-end device 11 can generate abstract machine-readable information of the test case, i.e., test case abstract representation information, according to the interactive input of the user and a preset test case mapping template, for example, the test case abstract representation information can be a json representation of the test case. That is, the interactive front-end device 11 can realize automatic writing of the test case according to the interactive input of the user. The interactive front-end device 11 can include but is not limited to a mobile phone, a computer, a tablet computer, and the like.

[0026] The test end device 12 can run a Web application and display a Web page, and test the Web application. For example, the test end device 12 can have a Web application such as a browser, but is not limited to this. The Web automation test system can include one or more test end devices 12. In the case where the Web automation test system includes multiple test end devices 12, the multiple test end devices 12 can include test end devices 12 with the same operating system and hardware components, or test end devices 12 with different operating systems and hardware components. The operating system of the test end device can include but is not limited to Linux, Windows, and the like. The test end device 12 can communicate with the interactive front end. The test end device 12 can be regarded as the driving end of the test. The test end device 12 can call a multi-modal large language model based on the test case abstract representation information generated by the interactive front-end device 11 to utilize the processing capability of the multi-modal large language model, especially the visual understanding capability, to perform reasoning, obtain reasoning result information capable of representing operations performed on the interactive elements in the Web page, convert the reasoning result information into machine executable instructions and execute them, so as to realize Web automation testing.

[0027] In some embodiments, the Web automation test system can further include a management end platform. FIG. 2 The architecture schematic diagram of the Web automation test system provided for another embodiment of the present application is as follows, FIG. 2 and FIG. 1The difference lies in that the Web automation testing system may also include a management platform 13. The management platform 13 can be implemented by a single device or multiple devices; the type and number of devices forming the management platform 13 are not limited here. The management platform 13 can communicate and interact with the interactive front-end device 11. After generating test case abstract representation information, the interactive front-end device 11 can upload the test case abstract representation information to the management platform 13. The management platform 13 can be used to manage the test case abstract representation information and distribute it to the test end device 12 that matches the test requirements. The management platform 13 can also communicate and interact with the test end device 12. The test results obtained by the test end device 12 after performing Web automation testing can also be uploaded to the management platform 13. The management platform 13 can store the test results for subsequent optimization and adjustment of the Web application based on the test results.

[0028] The web automation testing system can execute the web automation testing methods provided in this application, which will be described in detail below.

[0029] This application provides a Web automation testing method that can be applied to scenarios of automated testing of Web applications. This Web automation testing method can be executed by the Web automation testing system in the above embodiments. FIG. 3 A flowchart of a web automation testing method provided in an embodiment of this application is shown below. FIG. 3 As shown, the Web automation testing method may include steps S201 to S204.

[0030] In step S201, the interactive front end generates abstract representation information of test cases based on the user's interactive input and the preset test case mapping template.

[0031] The interactive front end is a front-end device capable of human-computer interaction with the user; this interactive front end is the interactive front end device 11 in the above embodiment. User input is used to realize human-computer interaction between the user and the interactive front end. The type of interactive input is not limited here; for example, interactive input may include, but is not limited to, click input, long press input, swipe input, text input, and voice input.

[0032] The test case mapping template is used to convert the information indicated by the interactive input into test case abstract representation information. The test case abstract representation information is machine-readable abstract representation information representing the test case, for example, the test case abstract representation information can be information in json format. The test case mapping template can include keys and values, the keys can include information labels, the values in the test case mapping template can be first set as null values, placeholder values or default values, after obtaining the information indicated by the interactive input, the information value corresponding to the information label can be extracted from the information indicated by the interactive input, and the information value is filled into the position of the value in the test case mapping template, so as to generate the test case abstract representation information. The test case abstract representation information includes the target test task type indicated by the interactive input and the intent natural language description information. The target test task type includes the test task type indicated by the interactive input. The intent natural language description information includes the natural language description of the test intent of the user input by the interactive input. The test case abstract representation information can represent which type of test task the user wants to use to realize the test corresponding to the test intent.

[0033] The test task types can be abstracted and divided in advance according to test tasks involved in traditional test framework code. In some examples, the test task types can include a page operation type and a tool invocation type. The test task of the page operation type includes a test task of operating a web page. The page operation type can include a long-range task type and a short-range task type. The long-range task type corresponds to a test task combined by multiple immediate operation actions, or in other words, the long-range task type corresponds to a task representing an autonomous planning action. An autonomous planning action can be regarded as an open action, which is an action that can be decomposed into multiple immediate operation actions according to a tested web page after planning, i.e., the open action can be combined by multiple immediate operation actions, which are obtained by planning and decomposing according to the tested web page. It should be noted that the tested web page corresponding to the test task of the long-range task type can be one or multiple. In the case where the tested web page corresponding to the test task of the long-range task type is multiple, the multiple immediate operation actions corresponding to the test task of the long-range task type can belong to different tested web pages. For example, after executing a certain immediate operation action, the tested web page jumps from one tested web page to another tested web page, and then executes the next immediate operation action. The test task of the short-range task type is a task including one immediate operation action, i.e., the test task of the short-range task type is used to execute one immediate operation action. The test task of the short-range task type can be a complementary task of the test task of the long-range task type. Through mutual complementation and cooperation of the test task of the long-range task type and the test task of the short-range task type, the robustness of the test case can be enhanced, more test scenarios can be adapted, and the test task of the short-range task type as a complement of the test task of the long-range task type can also make the web automation test more accurate. The immediate operation action is a single step that can be immediately executed, for example, the immediate operation action can include but is not limited to single click, double click, text input, scrolling, page assertion, certificate selection, page jump, and the like. The test task of the tool invocation type is a task implemented by calling an external tool. Specifically, the test task of the tool invocation type can call the external tool through a model context protocol (MCP). The external tool can include a database and other tools, and the other tools can include but are not limited to a service state checking tool, a data import tool, a cleaning tool, a mail notification tool, and the like.Correspondingly, the tool invocation type corresponding test task can include a database operation task and other tool operation tasks, the database operation task can include but is not limited to a database query task, a database execution task, a database assertion task and the like, and the other tool operation task can include but is not limited to a service state check task, a data import task, a cleaning task, a mail notification task and the like. The target test task type is the test task type indicated by the interactive input, which can include one or more than two of the above test task types. That is, the target test task type includes the page operation type, the page operation type includes the long-range task type and the short-range task type, the long-range task type corresponds to the task combined by multiple instant operation actions, and the short-range task type corresponds to the task including one instant operation action. The target test task type also includes the tool invocation type, and the test task corresponding to the tool invocation type is implemented by calling an external tool through a model context protocol.

[0034] In some examples, the interaction front end can display an interaction interface, the interaction interface being a visual interface for editing a test case, and the interaction front end can determine the test intention of the user through interaction input of the user on the interaction interface. The interaction interface can be preconfigured with atomic components according to the classification of test task types, each atomic component corresponding to a test task type, and the test task types are displayed in the form of atomic components on the interaction interface. In the case of selecting an atomic component to select a target test task type through interaction input, the intention natural language description information can also be input through interaction input. The interaction input can include component selection input and natural language description input. The component selection input is used to select an atomic component in the interaction interface. The natural language description input is used to input the intention natural language description information. The interaction front end can select one or more target test task types from the test task types supported by the atomic components in response to the component selection input of the user on the atomic components in the interaction interface; obtain the intention natural language description information indicated by the natural language description input in response to the natural language description input of the user on the interaction interface; and substitute the target test task types and the intention natural language description information into a test case mapping template to obtain test case abstract representation information. The interaction interface can include a plurality of atomic components, and one or more atomic components can be selected in the interaction interface through component selection input, and the test task types supported by the selected atomic components are determined as target task types. Alternatively, the first interaction interface includes a component addition control, and through the operation of the component addition control, the interaction interface can be switched to another interaction interface dedicated to displaying atomic components, and one or more atomic components can be selected through component selection input, and the test task types supported by the selected atomic components are determined as target task types. The interaction interface can also provide an input box to input the intention natural language description information through natural language description input. The target test task types obtained through the component selection input and the intention natural language description information obtained through the natural language description input can be substituted into the corresponding value positions of the test case mapping template to obtain the test case abstract representation information.

[0035] For example, the corresponding atomic component can be dragged through the component selection input in the interaction interface, and the intention natural language description information describing the test intention of the user can be described through the natural language description input. FIG. 4 An example of the interaction interface provided by the embodiments of the present application is shown in the schematic diagram as FIG. 4 The basic operation control can include FIG. 4The save control 211, the delete control 212, the add control 213, the upload control 214, the download control 215 and the execution control 216 in the test case name input box 210. The atomic component can include a planning component 221, an input component 222, a database component 223 and a more component 224. The planning component 221 is the atomic component corresponding to the long-term task type, the input component 222 is the atomic component corresponding to the input task type, the database component 223 is the atomic component corresponding to the database task type, and the more component 224 is the atomic component corresponding to other task types. The input box can include input box 231, input box 232, input box 233, input box 234 and input box 235. The test case name in the input box corresponding to the name, FIG. 4 The test case name in the input box corresponding to the name, FIG. 4 The test case name in the input box corresponding to the name, FIG. 4 The test case name in the input box corresponding to the name, FIG. 4 The test case name in the input box corresponding to the name, FIG. 4 The test case name in the input box corresponding to the name, FIG. 4 The test case name in the input box corresponding to the name, FIG. 5 The test case name in the input box corresponding to the name,

[0036]

[0037] Wherein, "params" represents parameters, "url" represents an access uniform resource locator, "tasks" represents test tasks, "flow" represents a data processing framework, "type" represents a test task type, "prompt" represents an intent natural language description information, and "target" represents test target information.

[0038] In step S202, the target screenshot of the tested Web page and the attribute information of the interactive element in the tested Web page are obtained by the test terminal based on the test case abstract representation information.

[0039] The test case abstract representation information can be parsed, and the test task represented by the test case abstract representation information can be decomposed to obtain corresponding instant operation actions, so as to obtain the target screenshot of the tested Web page and the attribute information of the interactive element in the tested Web page based on the instant operation actions. The target screenshot is a screenshot of the tested Web page, which can be a high-definition screenshot of a captured Web application such as a browser window. The target screenshot can completely represent the visual presentation of the tested Web page.

[0040] The interactive element in the tested Web page is an element that can be interacted with in the tested Web page, for example, the interactive element can include but is not limited to buttons, links, input boxes, drop-down menus, multi-line text input boxes, etc. The attribute information of the interactive element can embody the attributes of the interactive element, for example, the attribute information can include but is not limited to the identification of the interactive element, i.e. id, the element label, i.e. aria-label, the visual prompt text, i.e. placeholder, the visible text of the interactive element, the position information of the interactive element, etc. The target screenshot is visual data, and the attribute information of the interactive element is structured data. The visual data and the structured data can be fused into multi-modal data in a unified format to provide the multi-modal large language for processing.

[0041] In step S203, the test terminal calls the multi-modal large language model to perform reasoning based on the target screenshot, the attribute information of the interactive element, and the intent natural language description information, and outputs reasoning result information.

[0042] The multi-modal large language model can perform reasoning according to the input multi-modal data. The information input into the multi-modal large language model can include the target screenshot, the attribute information of the interactive element, and the intent natural language description information. The multi-modal large language model can utilize visual reasoning capability to process the target screenshot and the attribute information of the interactive element, and utilize language reasoning capability to process the intent natural language description information, to automatically locate the position of the interactive element and the rendering effect of the perceived tested Web page.

[0043] The multimodal large language model can perform inference for each instant operation action, and each instant operation action can correspond to a reasoning result information. In the case that the target test task type includes a long-range task type, since the test task corresponding to the long-range task type is composed of multiple instant operation actions, for a test task corresponding to a long-range task type, the multimodal large language model can perform multiple inferences to obtain multiple reasoning result information.

[0044] The reasoning result information can represent an operation performed on the target interactive element at the target position in the tested Web page. The intent natural language description information can embody the test intent of the user. According to the target screenshot, the attribute information of the interactive element, and the intent natural language description information, the multimodal large language model can infer that the test intent of the user is to perform a certain operation on an interactive element at a certain position in the tested Web page. The target interactive element is the interactive element intended to be operated in the test, and the target position is the position of the target interactive element in the tested Web page. The operation performed on the target interactive element is related to the target test task type. The reasoning result information can be machine-readable information, such as the format of the reasoning result information being json format. The multimodal large language model can infer the target position, the target interactive element, and the operation performed on the target interactive element based on the target screenshot, the attribute information of the interactive element, and the intent natural language description information without directly inputting the target interactive element and the target position. For different tested Web pages, the test end can infer the target position, the target interactive element, and the operation performed on the target interactive element using the multimodal large language model. Web automation testing is no longer limited by the differences in the user interfaces of the tested Web pages, improving the compatibility of Web automation testing and thus improving the stability of Web automation testing.

[0045] In step S204, the test end converts the reasoning result information into machine executable instructions, and executes the machine executable instructions to implement Web automation testing.

[0046] The reasoning result information is machine-readable information, but cannot be executed by a machine. The reasoning result information needs to be translated into machine executable instructions, and the test end executes the machine executable instructions to perform Web automation testing. For example, the reasoning result information represents an input operation at the target position, and the input value value is “mechanical keyboard”. After executing the machine executable instructions converted from the reasoning result information, the relevant API in the execution interface can be called to perform an input operation on the search box at the target position in the tested Web page, and “mechanical keyboard” is input into the search box.

[0047] In some examples, each instant operation action can correspond to one inference result information, each inference result information corresponds to one machine executable instruction, if the test task represented by the intention natural language description information includes multiple instant operation actions, the inferences can be performed in the order of the multiple instant operation actions in the test task, each time an inference result information is obtained, the inference result information is converted into a machine executable instruction, and the machine executable instruction is executed, until the machine executable instructions corresponding to the multiple instant operation actions in the test task are all executed, and the current Web automatic test is ended.

[0048] In the embodiments of the present application, the test case abstract representation information capable of representing the user test intention can be automatically generated according to the user's interactive input and the preset test case mapping template, the target screenshot of the Web page under test and the attribute information of the interactive element in the Web page under test are obtained based on the test case abstract representation information, and the multi-modal large language model is called to perform inference based on the multi-modal data of the target screenshot, the attribute information of the interactive element and the intention natural language description information, to obtain the inference result information capable of representing the operation performed on the target interactive element located at the target position in the Web page under test, so as to convert the inference result information into a machine executable instruction and execute it, thereby realizing Web automatic test. Without the need for the user to define the specific position of the interactive element to be operated, the visual understanding ability, natural language understanding ability and inference ability of the multi-modal large language model can be used to automatically locate the interactive element to be operated and the position of the interactive element in the test task, even if the user interface of the Web page under test is changed, the multi-modal large language model can accurately locate the interactive element to be operated and the position of the interactive element in the test task, without the need for the user to manually modify or re-write the test script, thereby improving the stability and efficiency of Web automatic test. Moreover, the test case abstract representation information is generated based on the user's interactive input, and the inference result information is obtained through the inference of the multi-modal large language model based on the intention natural language description information, and the process only needs the user to perform natural language description, without the need for the user to use professional programming language to write test script, thereby reducing the difficulty of Web automatic test, improving the efficiency of Web automatic test, enabling more users to participate in Web automatic test, and avoiding the occurrence of business islands and business flow bottlenecks in the Web business process.

[0049] In some embodiments, the step S202 can be specifically refined as: calling the large language model to decompose the test task indicated by the test case abstract representation information based on the test case abstract representation information, to obtain action information of the corresponding immediate operation action; and in response to the action information of the immediate operation action, obtaining attribute information of the target screenshot and the interactive element. A prompt word can be generated based on the test case abstract representation information, and the prompt word including the test case abstract representation information is input into the large language model, which instructs the large language model to decompose the test task indicated by the test case abstract representation information into an immediate operation action. The large language model can determine the target test task type according to the test case abstract representation information. If the target test task type includes a tool calling type, the MCP can be used to replace the external tool to implement the test task of the tool calling type. If the target test task type includes a page operation type, the large language model is called to decompose the test task. The test task indicated by the test case abstract representation information can include one immediate operation action or multiple immediate operation actions. If the type of the test task is a long-range task type, the test task of the long-range task type can be decomposed into multiple immediate operation actions. For example, the natural language description information of the test task corresponding to the long-range task type is “search for ‘mechanical keyboard’ on the current page, filter by ‘4 stars and above’, and add the first search result with a price lower than 100 yuan to the shopping cart”, which can be decomposed into three immediate operation actions: “1. Search for ‘mechanical keyboard’ on the current page”, “2. Filter by ‘4 stars and above’”, and “3. Add the first search result with a price lower than 100 yuan to the shopping cart”. If the type of the test task is a short-range task type, the decomposition of the test task of the short-range task type can obtain one immediate operation action. The action information of the immediate operation action is used to represent the immediate operation action and can include, but is not limited to, information for describing the immediate operation action. For each immediate operation action, the target screenshot of the tested Web page corresponding to the immediate operation action and the attribute information of the interactive element in the tested Web page can be obtained.

[0050] In some examples, for the decomposed immediate operation action, an enhanced operation such as parameter replacement can be required. The large language model can be called to decompose the test task indicated by the test case abstract representation information to obtain the action information of the immediate operation action; in the case where the action information of the immediate operation action includes to-be-enhanced information, the target information corresponding to the to-be-enhanced information is queried in the preset knowledge base, and the target information is used to enhance the action information of the immediate operation action. The action information of the immediate operation action includes to-be-enhanced information, indicating that the immediate operation action needs to be enhanced, and the preset knowledge base can be used to enhance the action information of the immediate operation action. The knowledge base stores domain knowledge information of the test business, which can provide business professional support for the immediate operation action decomposed from the test task. For example, the interactive interface for generating the test case abstract representation information is as followsFIG. 5 As shown in the figure, the name of the visited website input in the input box 233 corresponding to the visited uniform resource locator visitURL is “XX website”, and the web page to be visited during testing is the page of the XX website. The uniform resource locator (URL) corresponding to the XX website can be found in the preset knowledge base, and “XX website” in the action information of the instant operation action is replaced with the found URL, so that the web application can access the web page of the XX website, and the web page is used as the tested web page.

[0051] For example, The interactive element can represent a button. An example of a schematic diagram of a test end processing flow provided by an embodiment of the present application is shown in FIG. 6. The interactive element can represent a drop-down menu. As shown in the figure, the test end processing flow can include steps a1 to a9.

[0052] In step a1, test case abstract representation information is received.

[0053] In step a2, a large language model is called to perform test task decomposition according to the test case abstract representation information, and action information of an instant operation action is obtained.

[0054] In step a3, it is determined whether enhancement processing is needed. Specifically, it can be determined according to whether there is to-be-enhanced information in the action information. If there is to-be-enhanced information, it is determined that enhancement processing is needed; if there is no to-be-enhanced information, it is determined that enhancement processing is not needed. If enhancement processing is not needed, step a4 is executed; if enhancement processing is needed, step a5 is executed.

[0055] In step a4, the original action information of the instant operation action is maintained.

[0056] In step a5, target information is found in the knowledge base, and the target information is used to enhance the action information of the instant operation action.

[0057] In step a6, it is determined whether the test task is of a tool calling type. If the test task is of a tool calling type, step a7 is executed; if the test task is of a page operation type, i.e., not of a tool calling type, step a8 is executed.

[0058] In step a7, an external tool is called. For example, an external database is called to perform database query.

[0059] In step a8, the web application is interacted with. For example, Playwright can be used to interact with the target browser to obtain the required information. The specific operation in step a8 can be processed by calling a multi-modal large language model. For details, please refer to the description of the processing of the multi-modal large language model in the embodiments of the present application, which will not be repeated here.

[0060] In step a9, the machine executable instructions are generated and executed.

[0061] The specific content of steps a1 to a9 can be referred to the relevant description in the above embodiments, which will not be repeated here.

[0062] In some examples, the interaction element and the attribute information of the interaction element can be obtained by using the Document Object Model (DOM) information of the tested page. Specifically, the DOM information of the tested Web page can be parsed, and the interaction element and the attribute information of the interaction element can be extracted from the DOM information. The attribute information can include position information, which can include but is not limited to coordinate information of the interaction element in the tested Web page. The original DOM information of the tested Web page is large in quantity, and part of the DOM information is equivalent to noise information for the multi-modal large language model, which can reduce the processing efficiency of the multi-modal large language model. The DOM information can be distilled, that is, Distillation, to extract the interaction element and the attribute information of the interaction element from a large amount of DOM information, obtain core information, and reduce noise information entering the multi-modal large language model. For example, the interaction element and the attribute information of the interaction element can be extracted from the DOM information <button>、 、 <input> 、 <select>、< / select> <textarea>The interactive element can represent a multi-line text input box. The specific content of the attribute information of the interactive element can be referred to the related description in the above embodiments, which will not be described here again. By extracting the interactive element and the attribute information of the interactive element from the DOM information, a structured text summary concise, efficient and friendly to the multi-modal large language model can be generated. After extracting the interactive element and the attribute information of the interactive element from the DOM information, the target screenshot can be further processed to mark the interactive element position identifier on the target screenshot, so as to input the target screenshot marked with the interactive element position identifier into the multi-modal large language model to assist the multi-modal large language model in reasoning, and further improve the accuracy of the multi-modal large language model in reasoning. The interactive element position identifier can include but is not limited to a marking box and an identification label. The marking box can enclose the interactive element with a square or other shape box, and the identification label can identify different interactive elements to distinguish different interactive elements, and the identification labels of different interactive elements are different.< / textarea> < / button> <button> FIG. 6 may represent a connection, <input> may represent an input box, <select>FIG. 6< / select> <textarea>FIG. 7< / textarea>

[0063] For example, FIG. 6 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 7 FIG. 7 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 7 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 6 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 7 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 5 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 6 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 6 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 6 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 6 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 6 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 6 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 6 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 6 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 7 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 4 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 4 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34. FIG. 8 An example of a schematic diagram of a target screenshot of a tested Web page provided for an embodiment of the present application is shown in FIG. 17, which can include an input box 31, a search button 32, six tab buttons 331, 332, 333, 334, 335 and 336, and a drop-down button 34.

[0064] <input ElementId＝"41"type＝"text"id＝"twotabsearchtextbox"name＝"field-keywords"autocomplete＝"off"placeholder＝"Search XX"class＝".nav-input.nav-progressive-attribute"dir＝"auto"tabindex＝"0"aria-label＝"SearchXX"role＝"searchbox"aria-autocomplete＝"list"aria-controls＝"sac-autocomplete-results-container"aria-expanded＝"false"aria-haspopup＝"grid"spellcheck＝"false"left＝"374"top＝"11"width＝"617"height＝"38">

[0065] Search XX

[0066]

[0067] The "left=374 top=11 width=617 height=38" in the structured information including the attribute information of the interactive elements in the above is the position information of the input box corresponding to the digital tag 41.

[0068] ​By "distilling" the DOM information and using the attribute information in the DOM information to mark the interactive elements in the target screenshot, the multimodal large language model can obtain more information from the target screenshot and the attribute information of the interactive elements, and improve the accuracy of the multimodal large language model reasoning.

[0069] In some examples, the multimodal large language model can be used to obtain the interactive elements and the attribute information of the interactive elements. The multimodal large language model can be used to identify the interactive elements and the attribute information of the interactive elements from the target screenshot. The attribute information includes position information. If the multimodal large language model used has a visual positioning function, a bounding box or a bounding point can be generated in the target screenshot to accurately locate the interactive elements and provide stable attribute information output. Using the visual positioning function of the multimodal large language model can speed up the reasoning of the multimodal large language model, improve the efficiency of Web automated testing, and avoid interference of excessive DOM information with the multimodal large model in performing operations or locating interactive elements.

[0070] In some examples, the step S203 can be specifically refined as follows: generating a target prompt word based on the target screenshot, the attribute information of the interactive elements, and the natural language description information of the intent; and inputting the target prompt word into the multimodal large language model to perform reasoning to obtain the reasoning result information output by the multimodal large language model. If the attribute information of the interactive elements is obtained based on the DOM information of the Web page being tested, the target prompt word includes the target screenshot marked with the interactive element position identifier, and the interactive element position identifier is determined based on the DOM information. In the case where the attribute information of the interactive elements is obtained based on the DOM information, the target prompt word can include the target screenshot marked with the interactive element position identifier, the attribute information of the interactive elements, and the natural language description information of the intent. In the case where the attribute information of the interactive elements is obtained based on the visual positioning function of the multimodal large language model, the target prompt word can include the target screenshot and the natural language description information of the intent, and the attribute information of the interactive elements can be obtained during the process of inputting the target screenshot into the multimodal large language model for processing.

[0071] The multimodal large language model can be called to perform reasoning in the above examples in two modes, i.e., a first mode and a second mode. The multimodal large language model can be reasoned in either the first mode or the second mode. The first mode is a mode in which the test task is pre-decomposed, all instant operation actions included in the test task are obtained, and then the instant operation actions are reasoned and tested one by one. The second mode is a mode in which the test task is decomposed, reasoned, and tested simultaneously.

[0072] In some examples, when the mode of task decomposition is the first mode, the test task is decomposed into a plurality of immediate operation actions. According to the arrangement order of the plurality of immediate operation actions, the multimodal large language model is called for multiple times of reasoning. Each time of reasoning corresponds to an immediate operation action, and the target prompt word input to the multimodal large language model as each time of reasoning is determined based on the target screenshot corresponding to the immediate operation action, the attribute information of the interactive element corresponding to the immediate operation action, the action information of the immediate operation action, and the natural language description information of the intention.

[0073] For example, the interaction interface for generating the abstract representation information of the test case is as shown in FIG. 1. FIG. 8 As shown in FIG. 1, the test task can be decomposed into three immediate operation actions: "1. Search for ‘mechanical keyboard’ on the current page", "2. Filter by ‘4 stars and above’", and "3. Add the first search result with a price lower than 100 yuan to the shopping cart". The target prompt word can be generated according to the target screenshot corresponding to the first immediate operation action, the attribute information of the interactive element corresponding to the first immediate operation action, the action information of the first immediate operation action, and the natural language description information of the intention, the target prompt word is input to the multimodal large language model, the reasoning result information output by the multimodal large language model is obtained, the reasoning result information can be in json format, and the reasoning result information can be as shown in the following:

[0074]

[0075] The above reasoning result information indicates that the input operation is performed at the target position indicated by the position information "pos": [382, 14, 990, 45] or the target position indicated by the interactive element position identifier "ElementId": 39, and the input value "value" is "mechanical keyboard". The reasoning result information corresponding to the first immediate operation action is converted into machine executable instructions, and the machine executable instructions are executed. Then, the target prompt word is generated according to the target screenshot corresponding to the second immediate operation action, the attribute information of the interactive element corresponding to the second immediate operation action, the action information of the second immediate operation action, and the natural language description information of the intention, the target prompt word is input to the multimodal large language model, the reasoning result information output by the multimodal large language model is obtained, the reasoning result information corresponding to the second immediate operation action is converted into machine executable instructions, and the machine executable instructions are executed. Then, the target prompt word is generated according to the target screenshot corresponding to the third immediate operation action, the attribute information of the interactive element corresponding to the third immediate operation action, the action information of the third immediate operation action, and the natural language description information of the intention, the target prompt word is input to the multimodal large language model, the reasoning result information output by the multimodal large language model is obtained, the reasoning result information corresponding to the third immediate operation action is converted into machine executable instructions, and the machine executable instructions are executed.

[0076] In some examples, the intention natural language description information includes test target information; in a case where the task decomposition mode is the second mode, a first immediate operation action is obtained by decomposing the test task first. The multimodal large language model is called for multiple times of reasoning until the reasoning result information matches the test target information. Each time of reasoning corresponds to an immediate operation action, the reasoning result information output by the multimodal large language model in this time of reasoning includes action information of a next immediate operation action obtained by reasoning, and a target prompt word input into the multimodal large language model in this time of reasoning is determined based on a target screenshot corresponding to the immediate operation action, attribute information of an interactive element corresponding to the immediate operation action, action information of the immediate operation action, the reasoning result information obtained in a previous time of reasoning, and the intention natural language description information, and the target prompt word input into the multimodal large language model in the first time of reasoning is determined based on a target screenshot corresponding to the first immediate operation action, attribute information of an interactive element corresponding to the first immediate operation action, action information of the first immediate operation action, and the intention natural language description information.

[0077] The second mode can be a loop processing mode based on observation, decision, and action. The test task can be decomposed first to obtain only a first immediate operation action, a target prompt word is obtained based on a target screenshot corresponding to the first immediate operation action, attribute information of an interactive element corresponding to the first immediate operation action, action information of the first immediate operation action, and the intention natural language description information, the target prompt word is input into the multimodal large language model, reasoning result information input into the multimodal large language model is obtained, and in addition to including content representing an operation performed on a target interactive element located at a target position in a tested Web page corresponding to the first immediate operation action, the reasoning result information can also include a second immediate operation action predicted by reasoning. A target prompt word is obtained based on a target screenshot corresponding to the second immediate operation action, attribute information of an interactive element corresponding to the second immediate operation action, action information of the second immediate operation action, the reasoning result information obtained in the first time of reasoning, and the intention natural language description information, the target prompt word is input into the multimodal large language model, reasoning result information input into the multimodal large language model is obtained, and in addition to including content representing an operation performed on a target interactive element located at a target position in a tested Web page corresponding to the second immediate operation action, the reasoning result information corresponding to the second immediate operation action can also include a third immediate operation action predicted by reasoning. In this way, until the reasoning result information output by the multimodal large language model matches the test target information, i.e., the reasoning result information represents that the test target indicated by the test target information has been reached, the next time of reasoning can be performed.

[0078] For example, the interaction interface for generating the test case abstract representation information is as shown in FIG. 6. FIG. 9 As shown, the test task can be first decomposed to obtain a first immediate operation action "1. Search'mechanical keyboard' on the current page", according to the target screenshot corresponding to the immediate operation action, the attribute information of the interactive element corresponding to the immediate operation action, the action information of the immediate operation action and the intention natural language description information, a target prompt word is generated, the target prompt word is input into the multi-modal large language model, and the inference result information output by the multi-modal large language model can be as follows:

[0079]

[0080] Compared with the scheme adopting the first mode, the inference result information corresponding to the first immediate operation action adopting the second mode further includes the action information of the next immediate operation action "what_to_do_next":"click the search button", that is, the second immediate operation action is "click the search button". "more_action" can represent whether there are more immediate operation actions, if "more_action" is true, it indicates that there are more immediate operation actions, that is, the test target indicated by the test target information has not been reached; if "more_action" is false, it indicates that there are no more immediate operation actions, that is, the test target indicated by the test target information has been reached. The inference processes corresponding to the subsequent multiple immediate operation actions are similar to the inference process corresponding to the first immediate operation action, and the difference is that the target prompt words corresponding to the subsequent multiple immediate operation actions further include historical information, that is, the inference result information corresponding to the previous immediate operation action, and the inference processes corresponding to the subsequent multiple immediate operation actions are not described herein.

[0081] The second mode is a mode of circular processing based on observation, decision and action, which can fully utilize the inference ability of the multi-modal large language model and ensure that long-range tasks can be accurately decomposed and executed.

[0082] Regardless of the first mode or the second mode, the user does not need to use XPath, CSS Selector or other query languages to locate the interactive element to inform the multi-modal large language model of the interactive element to be located, but rather indicates the test task to be executed through the intention natural language description information, that is, the natural language description, and completes the positioning of the interactive element through the multi-modal large language model in combination with the test case abstract representation information and the multi-modal fusion information including visual data and structured data, thereby reducing the difficulty of Web automated testing and improving the stability of Web automated testing.

[0083] In some embodiments, the multimodal large language model can also be used to handle abnormal situations of the web page to be tested by using its visual understanding capability. Specifically, when the multimodal large language model identifies an abnormal situation of the web page to be tested according to the target screenshot, the inference result information further includes abnormality elimination action information, and the machine executable instructions corresponding to the abnormality elimination action information are executed to eliminate the abnormality of the web page to be tested. The multimodal large language model can identify the abnormal situation of the web page to be tested from the target screenshot by using its visual understanding capability. The abnormal situation of the web page to be tested can include, but is not limited to, a pop-up window appearing on the web page to be tested, the web page to be tested freezing, and the like. The abnormality elimination action information can be used to indicate an action to eliminate the abnormality. The action to eliminate the abnormality can include, but is not limited to, a waiting action, a closing abnormality action, a refreshing action, and the like. For example, the multimodal large language model identifies a pop-up window appearing on the web page to be tested from the target screenshot, and the inference result information output by the multimodal large language model after reasoning can include information indicating a waiting action or a closing pop-up window action. For another example, the multimodal large language model identifies that the web page to be tested freezes due to network fluctuations from the target screenshot, and the inference result information output by the multimodal large language model after reasoning can include information indicating a waiting action or a refreshing action.

[0084] In the web automation testing scenario, the multimodal large language model can be used to solve abnormal situations in the testing process, so that the web automation testing is more robust and has better robustness.

[0085] In the embodiments of the present application, a web automation testing report can also be generated according to the state of the web page to be tested and the inference result information of the multimodal large language model, and the web application can be detected and optimized according to the web automation testing report.

[0086] The test terminal in the above embodiments can be composed of multiple modules. FIG. 9 An example structure diagram of a test terminal provided in the embodiments of the present application is shown in FIG. 5. FIG. 9 As shown in FIG. 5, the test terminal 500 can include an input processor 501, a state awareness module 502, a multimodal state engine 503, a multimodal large language model inference core 504, an action translation layer 505, an execution interface 506, a memory module 507, and a knowledge base 508.

[0087] The input processor 501 can receive and parse the test case abstract representation information generated according to the user interaction input, perform decomposition of the test task, and call external tools. The input processor 501 can call a large language model to perform decomposition of the test task.

[0088] The state perception module 502 can interact with the web application under test to obtain raw data required for constructing the multi-modal data, including the target screenshot of the web page under test and the attribute information of the interactive elements in the web page under test. The attribute information of the interactive elements can be obtained from the DOM information. The state perception module 502 can implement the above functions by using the Playwright API, JavaScript, etc.

[0089] The multi-modal state engine 503 can fuse the raw data to obtain the multi-modal data, and can also implement the "distillation" of the DOM information and the marking of the interactive element position identification in the above embodiments. The multi-modal state engine 503 can implement the above functions by means of DOM parsing and cleaning algorithms, image processing, etc.

[0090] The multi-modal large language model inference core 504 can take the multi-modal data, the intent natural language description information, the historical information, etc. as the input of the multi-modal large language model, and infer and generate the inference result information. The multi-modal large language model inference core 504 can implement the above functions by means of the multi-modal large language model, reactive planning prompt engineering, etc.

[0091] The action translation layer 505 can parse and translate the inference result information into machine executable instructions executable by the underlying browser framework. The action translation layer 505 can implement the above functions by means of json parsing, command mapping, etc.

[0092] The execution interface 506 can call the API of the underlying automation framework, and accurately execute the executable instructions obtained after translation in the web application such as the browser. The execution interface 506 can implement the above functions by means of the Playwright API, etc.

[0093] The memory module 507 can store the action information of the current instant operation action, the corresponding inference result information and the historical information, to provide context information for subsequent inference decision, support long-term planning and error response. The memory module 507 can implement the above functions by means of short-term and long-term memory structure, vector database, etc.

[0094] The knowledge base 508 can store general knowledge information in the business field, to enhance the understanding ability of the large language model and the multi-modal large language model for web testing. The knowledge base 508 can implement the above functions by means of retrieval augmented generation, etc.

[0095] The application also provides a web automation testing device. ​ The structure schematic diagram of the web automation testing device provided by an embodiment of the application is shown in ​ As shown, the Web automation testing device 600 comprises a memory 601, a processor 602 and a computer program stored in the memory 601 and executable on the processor 602.

[0096] In some examples, the processor 602 described above can comprise a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured to implement one or more integrated circuits configured to implement embodiments of the present application.

[0097] The memory 601 can comprise a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory comprises one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software comprising computer-executable instructions and when the software is executed (e.g., by one or more processors), it is operable to perform operations described with reference to the Web automation testing method according to embodiments of the present application.

[0098] The processor 602 runs a computer program corresponding to the executable program code stored in the memory 601 by reading the executable program code, for implementing the Web automation testing method in the above-described embodiments.

[0099] In some examples, the Web automation testing device 600 can further comprise a communication interface 603 and a bus 604. Wherein, as ​ shown, the memory 601, the processor 602 and the communication interface 603 are connected through the bus 604 and complete communication among each other.

[0100] The communication interface 603 is mainly used to realize the communication among the modules, devices, units and / or devices in the embodiments of the present application. The input device and / or the output device can also be accessed through the communication interface 603.

[0101] Bus 604 includes a hardware, software, or both that couples components of web automation testing device 600 to each other. As an example and not by way of limitation, bus 604 can include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front-side bus (FSB), a Hyper Transport (HT) interconnect, an industry standard architecture (ISA) bus, an InfiniBand™ interconnect, a low pin count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a serial advanced technology attachment (SATA) bus, a video electronics standards board (VLB) bus, or another suitable bus or a combination of two or more of these. Where suitable, bus 604 can include one or more buses. Although this embodiment describes and shows a particular bus, this application contemplates any suitable bus or interconnect.

[0102] The application further provides a computer readable storage medium, which has stored computer program instructions, and the computer program instructions are executed by a processor to implement the web automation testing method in the above embodiments and achieve the same technical effects. To avoid repetition, details are not described here. The computer readable storage medium can include a non-transitory computer readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like, which is not limited here.

[0103] The application further provides a computer program product, which can include a computer program, and the computer program is executed by a processor to implement the web automation testing method in the above embodiments and achieve the same technical effects. To avoid repetition, details are not described here.

[0104] It should be clarified that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. For device embodiments, system embodiments, computer-readable storage medium embodiments, and computer program product embodiments, relevant parts can be referred to the description section of the method embodiments. This application is not limited to the specific steps and structures described above and shown in the figures. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application. Furthermore, for the sake of brevity, detailed descriptions of known methods and techniques are omitted here.

[0105] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0106] Those skilled in the art will understand that the above embodiments are exemplary and not restrictive. Different technical features appearing in different embodiments can be combined to achieve beneficial effects. Based on a study of the drawings, specification, and claims, those skilled in the art should be able to understand and implement other variations of the disclosed embodiments. In the claims, the term "comprising" does not exclude other means or steps; the quantifier "a" does not exclude a plurality; the terms "first" and "second" are used to identify names and not to indicate any particular order. Any reference numerals in the claims should not be construed as limiting the scope of protection. The functionality of multiple parts appearing in the claims can be implemented by a single hardware or software module. The appearance of certain technical features in different dependent claims does not mean that these technical features cannot be combined to achieve beneficial effects.< / button>

Claims

1. A method of web automation testing, characterized by, The method comprises the following steps: generating test case abstract representation information by an interactive front end according to user interaction input and a preset test case mapping template, the test case abstract representation information including target test task types indicated by the interaction input and intent natural language description information; obtaining a target screenshot of a tested Web page and attribute information of an interactive element in the tested Web page based on the test case abstract representation information by a test end; calling a multi-modal large language model based on the target screenshot, the attribute information of the interactive element and the intent natural language description information by the test end to perform reasoning and output reasoning result information, the reasoning result information representing an operation performed on a target interactive element located at a target position in the tested Web page; and converting the reasoning result information into machine executable instructions by the test end and executing the machine executable instructions to realize Web automated testing.

2. The method of claim 1, wherein, The interaction input includes component selection input and natural language description input. The method of generating test case abstract representation information according to user interaction input and a preset test case mapping template comprises the following steps: selecting one or more target test task types from among test task types supported by a plurality of atomic components in response to the component selection input of the user on the plurality of atomic components in the interactive interface; obtaining the intent natural language description information indicated by the natural language description input of the user on the interactive interface in response to the natural language description input of the user on the interactive interface; substituting the target test task types and the intent natural language description information into the test case mapping template to obtain the test case abstract representation information.

3. The method of claim 1, wherein, The method of obtaining a target screenshot of a tested Web page and attribute information of an interactive element in the tested Web page based on the test case abstract representation information comprises the following steps: calling a large language model to decompose a test task indicated by the test case abstract representation information based on the test case abstract representation information to obtain action information of a corresponding immediate operation action; obtaining the target screenshot and the attribute information of the interactive element in response to the action information of the immediate operation action.

4. The method of claim 3, wherein, The method of calling a large language model to decompose a test task indicated by the test case abstract representation information based on the test case abstract representation information to obtain action information of a corresponding immediate operation action comprises the following steps: calling the large language model to decompose the test task indicated by the test case abstract representation information to obtain action information of an immediate operation action; in the case where the action information of the immediate operation action includes to-be-enhanced information, querying a preset knowledge base for target information corresponding to the to-be-enhanced information and enhancing the action information of the immediate operation action using the target information.

5. The method of claim 1, wherein, The method of obtaining attribute information of the interactive element comprises the following steps: parsing document object model (DOM) information of the tested Web page and extracting the interactive element and the attribute information of the interactive element from the DOM information, the attribute information including position information; or, The visual positioning function of the multi-modal large language model is used to identify the interactive element and attribute information of the interactive element from the target screenshot, and the attribute information includes position information.

6. The method of claim 1, wherein, The multi-modal large language model is called to perform reasoning based on the target screenshot, the attribute information of the interactive element, and the intent natural language description information, and output reasoning result information, including: A target prompt word is generated based on the target screenshot, the attribute information of the interactive element, and the intent natural language description information; The target prompt word is input into the multi-modal large language model for reasoning to obtain the reasoning result information output by the multi-modal large language model; If the attribute information of the interactive element is obtained based on document object model (DOM) information of the tested Web page, the target prompt word includes the target screenshot marked with an interactive element position identifier, and the interactive element position identifier is determined according to the DOM information.

7. The method of claim 3, wherein, In the case of the first mode of task decomposition, the test task is decomposed to obtain a plurality of instant operation actions; The multi-modal large language model is called to perform reasoning based on the target screenshot, the attribute information of the interactive element, and the intent natural language description information, and output reasoning result information, including: The multi-modal large language model is called to perform a plurality of times of reasoning in the arrangement order of the plurality of instant operation actions, Each time of reasoning corresponds to an instant operation action, and the target prompt word input into the multi-modal large language model each time of reasoning is determined based on the target screenshot corresponding to the instant operation action, the attribute information of the interactive element corresponding to the instant operation action, action information of the instant operation action, and the intent natural language description information.

8. The method of claim 3, wherein, The intent natural language description information includes test target information; in the case of the second mode of task decomposition, the test task is first decomposed to obtain a first instant operation action; The multi-modal large language model is called to perform reasoning based on the target screenshot, the attribute information of the interactive element, and the intent natural language description information, and output reasoning result information, including: The multi-modal large language model is called to perform a plurality of times of reasoning until the reasoning result information matches the test target information; Each time of reasoning corresponds to an instant operation action, the reasoning result information output by the multi-modal large language model this time of reasoning includes action information of a next instant operation action obtained by reasoning, the target prompt word input into the multi-modal large language model this time of reasoning is determined based on the target screenshot corresponding to the instant operation action, the attribute information of the interactive element corresponding to the instant operation action, the action information of the instant operation action, the reasoning result information obtained by the last time of reasoning, and the intent natural language description information, and the target prompt word input into the multi-modal large language model the first time of reasoning is determined based on the target screenshot corresponding to the first instant operation action, the attribute information of the interactive element corresponding to the first instant operation action, the action information of the first instant operation action, and the intent natural language description information.

9. The method of claim 1, wherein, In a case where the multi-modal large language model is called to identify that the tested Web page has an exception according to the target screenshot, the inference result information further includes exception elimination action information, and the machine executable instruction corresponding to the exception elimination action information is executed to eliminate the exception of the tested Web page.

10. The method according to any one of claims 1 to 9, characterized in that, The target test task type includes a page operation type, and the page operation type includes a long-range task type and a short-range task type. The long-range task type corresponds to a task composed of multiple instant operation actions, and the short-range task type corresponds to a task including one instant operation action. The target test task type further includes a tool calling type, and the test task of the tool calling type is implemented by calling an external tool through a model context protocol.

11. A web automation testing system characterized by, Comprise: An interactive front-end device configured to generate test case abstract representation information according to user interaction input and a preset test case mapping template, wherein the test case abstract representation information includes a target test task type indicated by the interaction input and intent natural language description information; A test terminal device communicatively connected to the interactive front-end device and configured to obtain a target screenshot of a tested Web page and attribute information of an interactive element in the tested Web page based on the test case abstract representation information; and configured to call a multi-modal large language model to perform inference based on the target screenshot, the attribute information of the interactive element, and the intent natural language description information, and output inference result information representing an operation performed on a target interactive element located at a target position in the tested Web page; and configured to convert the inference result information into machine executable instructions and execute the machine executable instructions to implement Web automated testing.

12. The system of claim 11, wherein, Further comprise: A management terminal platform, wherein the test terminal device is communicatively connected to the test terminal device through the management terminal platform, and the management terminal platform is configured to manage the test case abstract representation information and issue the test case abstract representation information to the test terminal device matching a test requirement.

13. A web automation testing device, characterized by, Comprise: A processor and a memory storing computer program instructions; The processor executes the computer program instructions to implement the Web automated testing method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions, and the computer program instructions are executed by the processor to implement the Web automated testing method according to any one of claims 1 to 10.

15. A computer program product, characterised in that, The computer program is executed by the processor to implement the Web automated testing method according to any one of claims 1 to 10.

Images