Web interface element identification method, system, equipment and medium

Abstract

The invention discloses a Web interface element recognition method, system and device and a medium, belongs to the technical field of artificial intelligence and Web interface development, and aims to solve the technical problem of how to improve the recognition rate of dynamic elements and CSS hidden elements so as to enable Web interface element positioning to be more accurate. Performing screenshot by using a playwright screenshot function, converting the screenshot into a Base64 coded character string by using a function after the screenshot is completed, inputting the character string and a url into an intelligent agent, calling a DeepSeek-VL2 visual model by the intelligent agent, analyzing the screenshot and outputting identified structured element data; dOM structure analysis: inputting a page url address, and generating DOM structure data by using a DOM analysis function; dOM tree analysis and visual element matching: calling an element matching function, performing visual element and DOM structure matching, and obtaining all successfully matched element data; interface element extraction and positioning generation; and element application.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and web interface development technology, specifically a method, system, device, and medium for web interface element recognition. Background Technology

[0002] In modern web development and automated testing, accurately identifying web interface elements is a fundamental prerequisite for implementing many functionalities. As web applications become increasingly complex and dynamic, traditional web automation scripts based on fixed CSS / XPath selectors face significant challenges. These selectors typically rely on static paths or specific attributes of elements in the Document Object Model (DOM), making them highly susceptible to failure due to minor changes in page structure, layout, attributes, or asynchronous content loading. This results in fragile automation scripts, high maintenance costs, and poor adaptability.

[0003] While existing DOM parsing technologies can provide complete DOM snapshots and layout information, they cannot perceive the actual rendering effect of the page.

[0004] In summary, traditional web automation scripts suffer from drawbacks such as reliance on fixed selectors, difficulty in adapting to dynamic page changes, and high maintenance costs. Therefore, improving the recognition rate of dynamic elements and hidden CSS elements, thereby enabling more accurate positioning of web interface elements, is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] The technical objective of this invention is to provide a method, system, device, and medium for identifying web interface elements, in order to solve the problem of how to improve the recognition rate of dynamic elements and CSS hidden elements, thereby making the positioning of web interface elements more accurate.

[0006] The technical objective of this invention is achieved as follows: a method for identifying Web interface elements, the specific method of which is as follows:

[0007] Visual localization: Input the page URL address, take a screenshot using the Playwright screenshot function, convert the screenshot to a Base64 encoded string using a function, and input the string and URL into the agent. The agent calls the DeepSeek-VL2 visual model to analyze the screenshot and output the identified structured element data.

[0008] DOM structure parsing: Input the page URL address, use the DOM parsing function to generate DOM structure data;

[0009] DOM tree parsing and visual element matching: Call the element matching function to match visual elements and DOM structure, obtain the data of all successfully matched elements, and the elements that fail to match are pure visual elements;

[0010] Interface element extraction and location generation: Based on the interface element extraction strategy, obtain semantic location structure or hybrid location structure data for all successfully matched elements;

[0011] Element application: Apply the generated structured data to the automated test script, and run the script to test the successful execution of the automated script.

[0012] As a preferred option, visual positioning is as follows:

[0013] Construct an agent based on a DeepSeek-VL2 vision model, leveraging the powerful visual understanding capabilities of the DeepSeek-VL2 model to extract key visual features from images and output structured results;

[0014] Use the browser automation tool Playwright to open the test page, capture a screenshot of the test page rendering, convert the screenshot file into a Base64 encoded string, and store it in the BrowserState.screenshot property;

[0015] Create a request body that conforms to the requirements of the visual model API;

[0016] Use a POST request to invoke the agent of the DeepSeek-VL2 vision model, parse the structured results output by the agent, and generate structured element data in JSON format.

[0017] As a preferred approach, the DOM structure parsing is as follows:

[0018] Complete the basic DOM tree modeling: Define a parent class DOMBaseNode and two subclasses DOMTextNode and DOMBaseNode. The parent class DOMBaseNode is the base class for all DOM nodes, containing properties indicating whether a node is visible (is_visible) and its parent node (parent). The subclass DOMTextNode inherits from the parent class DOMBaseNode, representing text nodes. The information of a text node includes its text content (text) and node type (type, fixed as 'TEXT_NODE'), and has a method to check if there is a highlighted index in the parent node chain (has_parent_with_highlight_index). The subclass DOMElementNode inherits from the parent class DOMBaseNode, representing element nodes. The information of an element node includes its tag name (tag_name), XPath path (xpath), attribute dictionary (attributes), child node list (children) property, and indicators of whether it is interactive and whether it is a top-level element, as well as coordinate information. The element node information also includes a method for generating a string representation (__repr__) and a method for calculating a hash value (hash).

[0019] The DOM tree reconstruction function converts the page information corresponding to the URL into DOM structure data. This function initializes three parameters: `doHighlightElements`, `focusHighlightIndex`, and `focusHighlightIndex`, adding an element highlighting mechanism to mark important elements. It also defines several auxiliary functions (such as XPath path generation, element visibility determination, and element interactivity detection) to parse element types, element levels, attributes, and event binding relationships. Furthermore, the function handles special content (Shadow DOM, iframes) beyond ordinary elements. `doHighlightElements` indicates whether element highlighting is enabled; `focusHighlightIndex` represents the index of the highlighted element to be focused; and `viewportExpansion` represents the viewport expansion range value.

[0020] After the DOM tree is parsed, a structured output is performed, which includes tagName, attributes, xpath, and attributes of the children element.

[0021] As a preferred approach, DOM tree parsing and visual element matching are as follows:

[0022] The visual elements are processed by converting them into standardized element objects, extracting key features and performing the transformation. After the transformation is complete, a unique identifier is added and an element hash signature is generated. The key features include the element viewport size and scrolling status, precise spatial coordinate information, and advanced CSS selectors.

[0023] Process DOM nodes by adding unique identifiers and generating element hash signatures;

[0024] In the DOM structure, recursively traverse all child nodes starting from the root node and perform hash comparisons:

[0025] If the core attributes are consistent, the spatial positions overlap, and the structural paths are similar, then the match is considered successful, and the DOM node object is marked as an element that can be automatically manipulated.

[0026] If any one of the three requirements—identical core attributes, overlapping spatial locations, and similar structural paths—is not met, the match is considered to have failed, the DOM node object is marked as a pure visual element, and the process continues to traverse the next node.

[0027] More specifically, the extraction and positioning of interface elements are generated as follows:

[0028] After element matching is completed, a two-level path generation strategy is adopted, providing two types of interface element extraction methods. Testers can flexibly choose according to the actual situation of the project to extract elements that can be automated, as follows:

[0029] Level 1 is semantic positioning, which is suitable for scenarios where the function of an element is stable and unchanged. The generated example is represented as: {role:"shopping cart icon"};

[0030] Level 2 is a hybrid positioning, suitable for scenarios where element text is fine-tuned. The generated example is: {text:"Settlement",xpath:" / / div[@class='footer'] / button"}.

[0031] A web interface element recognition system, the system comprising:

[0032] The visual positioning module is used to input the page URL address, take a screenshot using the Playwright screenshot function, convert the screenshot to a Base64 encoded string using a function, and input the string and URL into the agent. The agent calls the DeepSeek-VL2 visual model to analyze the screenshot and output the recognized structured element data.

[0033] The parsing module is used to take a page URL as input and use DOM parsing functions to generate DOM structure data;

[0034] The matching module is used to call the element matching function to match visual elements and DOM structure, and to get the data of all successfully matched elements. Elements that fail to match are purely visual elements.

[0035] The element extraction module is used to extract semantic positioning structure or hybrid positioning structure data from all successfully matched elements according to the interface element extraction strategy.

[0036] The element application module is used to apply the generated structured data to automated test scripts and run the scripts to test the successful execution of the automated scripts.

[0037] Preferably, the visual positioning module includes:

[0038] The feature extraction submodule is used to build an agent of the DeepSeek-VL2 vision model. It utilizes the powerful visual understanding capabilities of the DeepSeek-VL2 model to extract key visual features from the image and output structured results.

[0039] The conversion submodule is used to open the test page using the browser automation tool Playwright, obtain a screenshot of the test page rendering, and convert the screenshot file into a Base64 encoded string, which is then stored in the BrowserState.screenshot property;

[0040] The request body creation submodule is used to create request bodies that conform to the requirements of the visual model API;

[0041] The parsing and generation submodule is used to call the agent of the DeepSeek-VL2 vision model using a POST request, and parse the structured results output by the agent to generate structured element data in JSON format;

[0042] The parsing module includes:

[0043] The DOM tree basic modeling submodule defines a parent class DOMBaseNode and two subclasses DOMTextNode. The parent class DOMBaseNode is the base class for all DOM nodes, containing properties indicating whether a node is visible (is_visible) and its parent node (parent). The subclass DOMTextNode inherits from DOMBaseNode and represents text nodes. Text node information includes text content (text) and node type (type, fixed at 'TEXT_NODE'), and a method to check if there is a highlighted index in the parent node chain (has_parent_with_highlight_index). The subclass DOMElementNode inherits from DOMBaseNode and represents element nodes. Element node information includes tag name (tag_name), XPath path (xpath), attribute dictionary (attributes), child node list (children) property, and indicators of interactivity and whether it is a top-level element, as well as coordinate information. Element node information also includes a method for generating a string representation (__repr__) and a method for calculating a hash value (hash).

[0044] The function reconstruction submodule is used to reconstruct the page information corresponding to the URL through the DOM tree, converting it into DOM structure data. The DOM tree reconstruction function initializes three parameters: `doHighlightElements`, `focusHighlightIndex`, and `focusHighlightIndex`, adding an element highlighting mechanism to mark important elements. It also defines several auxiliary functions (such as XPath path generation, element visibility determination, and element interactivity detection) to parse element types, element levels, attributes, and event binding relationships. The DOM tree reconstruction function also handles special content (Shadow DOM, iframe) other than ordinary elements. `doHighlightElements` indicates whether element highlighting is enabled; `focusHighlightIndex` represents the index of the highlighted element to be focused; and `viewportExpansion` represents the viewport expansion range value.

[0045] The structured output submodule is used to perform structured output after the DOM tree is parsed. The structured output includes tagName, attributes, xpath, and attributes of children elements.

[0046] More preferably, the matching module includes:

[0047] The key feature extraction submodule is used to process visual elements, convert visually recognized elements into standardized element objects, extract key features and perform the conversion, add a unique identifier and generate an element hash signature after the conversion is completed; among them, key features include element viewport size and scroll state, precise spatial coordinate information and advanced CSS selectors;

[0048] The hash signature generation submodule is used to process DOM nodes, add a unique identifier to the DOM node and generate an element hash signature;

[0049] The hash comparison submodule is used to recursively traverse all child nodes in the DOM structure, starting from the root node, and perform hash comparisons.

[0050] If the core attributes are consistent, the spatial positions overlap, and the structural paths are similar, then the match is considered successful, and the DOM node object is marked as an element that can be automatically manipulated.

[0051] If any one of the three requirements—identical core attributes, overlapping spatial locations, and similar structural paths—is not met, the match is considered to have failed, the DOM node object is marked as a pure visual element, and the process continues to traverse the next node.

[0052] The element extraction module employs a two-level path generation strategy, providing two methods for extracting UI elements. Testers can flexibly choose the method based on the actual project situation to extract elements that can be automated, as detailed below:

[0053] Level 1 is semantic positioning, which is suitable for scenarios where the function of an element is stable and unchanged. The generated example is represented as: {role:"shopping cart icon"};

[0054] Level 2 is a hybrid positioning, suitable for scenarios where element text is fine-tuned. The generated example is: {text:"Settlement",xpath:" / / div[@class='footer'] / button"}.

[0055] An electronic device includes: a memory and at least one processor;

[0056] The memory contains computer programs;

[0057] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the Web interface element recognition method as described above.

[0058] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the Web interface element recognition method described above.

[0059] The Web interface element recognition method, system, device, and medium of the present invention have the following advantages:

[0060] (i) This invention utilizes the multimodal cognitive capabilities of the DeepSeek-VL2 model to accurately identify Web interface elements through comparison of visual positioning and DOM parsing. This solves the problems of traditional Web automation scripts relying on fixed selectors, being difficult to adapt to dynamic page changes, and having high maintenance costs. At the same time, the interface element extraction module automatically generates element positioning paths, eliminating the need for manual writing, which greatly shortens the writing time of automated test scripts and improves the efficiency and accuracy of automated testing.

[0061] (ii) Based on the original DOM parsing technology, this invention creatively introduces the DeepSeek-VL2 visual model. Through the collaborative matching of visual positioning and DOM parsing dual modes, it improves the recognition rate of dynamic elements and CSS hidden elements, thereby making the positioning of web interface elements more accurate.

[0062] (III) This invention breaks away from the reliance on manual element annotation in previous automated testing methods. Through multimodal automatic annotation, it improves the coverage of page element recognition. Compared to the pure DOM solution, this patent is more accurate in recognizing dynamic ID elements and CSS hidden elements. Attached Figure Description

[0063] The invention will be further described below with reference to the accompanying drawings.

[0064] Appendix Figure 1 A flowchart of a method for identifying web interface elements. Detailed Implementation

[0065] The Web interface element recognition method, system, device, and medium of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] Example 1:

[0067] As attached Figure 1 As shown in the figure, this embodiment provides a method for identifying web interface elements, which is as follows:

[0068] S1. Visual localization: Input the page URL address, take a screenshot using the Playwright screenshot function, convert the screenshot to a Base64 encoded string using a function, and input the string and URL into the agent. The agent calls the DeepSeek-VL2 visual model to analyze the screenshot and output the recognized structured element data.

[0069] S2, DOM Structure Parsing: Input the page URL address, use the DOM parsing function to generate DOM structure data;

[0070] S3, DOM tree parsing and visual element matching: Call the element matching function to match visual elements and DOM structure, obtain the data of all successfully matched elements, and the elements that fail to match are pure visual elements;

[0071] S4. Interface element extraction and location generation: Based on the interface element extraction strategy, obtain semantic location structure or hybrid location structure data for all successfully matched elements.

[0072] S5. Element Application: Apply the generated structured data to the automated test script and run the script to test the successful execution of the automated script.

[0073] The visual positioning in step S1 of this embodiment is as follows:

[0074] S101. Construct an agent based on a DeepSeek-VL2 visual model, and use the powerful visual understanding capabilities of the DeepSeek-VL2 model to extract key visual features from the image and output structured results.

[0075] S102. Use the browser automation tool Playwright to open the test page, obtain a screenshot of the test page rendering, and convert the screenshot file into a Base64 encoded string and store it in the BrowserState.screenshot property;

[0076] S103. Create a request body that conforms to the requirements of the visual model API;

[0077] S104. Use a POST request to call the agent of the DeepSeek-VL2 vision model, and parse the structured results output by the agent to generate structured element data in JSON format, as shown in the example below:

[0078]

[0079]

[0080] The DOM structure parsing in step S2 of this embodiment is as follows:

[0081] S201. Complete the basic DOM tree modeling: Define a parent class DOMBaseNode and two subclasses DOMTextNode. The parent class DOMBaseNode is the base class for all DOM nodes, containing attributes for node visibility (is_visible) and parent node (parent). The subclass DOMTextNode inherits from the parent class DOMBaseNode, representing text nodes. The information of a text node includes text content (text) and node type (type, fixed as 'TEXT_NODE'), and has a method to check if there is a highlighted index in the parent node chain (has_parent_with_highlight_index). The subclass DOMElementNode inherits from the parent class DOMBaseNode, representing element nodes. The information of an element node includes tag name (tag_name), XPath path (xpath), attribute dictionary (attributes), child node list (children) attribute, and indicators of whether it is interactive and whether it is a top-level element, as well as coordinate information. The element node information also includes a method for generating a string representation (__repr__) and a method for calculating a hash value (hash).

[0082] S202. The DOM tree reconstruction function converts the page information corresponding to the URL into DOM structure data. This function initializes three parameters: `doHighlightElements`, `focusHighlightIndex`, and `focusHighlightIndex`, adding an element highlighting mechanism to mark important elements. It also defines several auxiliary functions (such as XPath path generation, element visibility determination, and element interactivity detection) to parse element types, element levels, attributes, and event binding relationships. Furthermore, the function handles special content (Shadow DOM, iframe) other than ordinary elements. `doHighlightElements` indicates whether element highlighting is enabled; `focusHighlightIndex` represents the index of the highlighted element to be focused; and `viewportExpansion` represents the viewport expansion range value.

[0083] S203. After completing the DOM tree parsing, perform structured output. The structured output includes tagName, attributes, xpath, and attributes of the children elements; an example is shown below:

[0084]

[0085] The DOM tree parsing and visual element matching in step S3 of this embodiment are as follows:

[0086] S301. Process the visual elements, convert the visually recognized elements into standardized element objects, extract key features and perform the conversion, add a unique identifier and generate an element hash signature after the conversion is completed; among them, the key features include the element viewport size and scrolling state, precise spatial coordinate information and advanced CSS selectors;

[0087] S302. Process the DOM nodes, add a unique identifier to the DOM nodes and generate element hash signatures;

[0088] S303. In the DOM structure, recursively traverse all child nodes starting from the root node and perform hash comparisons:

[0089] ① If the core attributes are consistent, the spatial positions overlap, and the structural paths are similar, then the match is considered successful, and the DOM node object is marked as an element that can be automatically manipulated;

[0090] ② If any one of the three requirements—identical core attributes, overlapping spatial locations, or similar structural paths—is not met, the match is considered to have failed, the DOM node object is marked as a purely visual element, and the process continues to traverse the next node.

[0091] The specific steps for extracting and locating interface elements in step S4 of this embodiment are as follows:

[0092] After element matching is completed, a two-level path generation strategy is adopted, providing two types of interface element extraction methods. Testers can flexibly choose according to the actual situation of the project to extract elements that can be automated, as follows:

[0093] Level 1 is semantic positioning, which is suitable for scenarios where the function of an element is stable and unchanged. The generated example is represented as: {role:"shopping cart icon"};

[0094] Level 2 is a hybrid positioning, suitable for scenarios where element text is fine-tuned. The generated example is: {text:"Settlement",xpath:" / / div[@class='footer'] / button"}.

[0095] The following is sample code for the dynamic path synthesis algorithm:

[0096]

[0097] The specific environment configuration for this embodiment is as follows:

[0098] (1) Install Playwright: Run the command npm install playwright in the project directory to install the Playwright testing framework.

[0099] (2) Configure large model: Call the DeepSeek-VL2 large model API and configure OPENAI_ENDPOINT and OPENAI_API_KEY in the settings file.

[0100] OPENAI_ENDPOINT=https: / / api.deepseek.com / vl2 / ui_analysis

[0101] OPENAI_API_KEY=*******;

[0102] (3) Constructing the intelligent agent: The output content is in JSON format and is output according to the attributes.

[0103] The complete workflow example of this embodiment is as follows:

[0104]

[0105]

[0106] Example 2:

[0107] This embodiment provides a web interface element recognition system, which includes:

[0108] The visual positioning module is used to input the page URL address, take a screenshot using the Playwright screenshot function, convert the screenshot to a Base64 encoded string using a function, and input the string and URL into the agent. The agent calls the DeepSeek-VL2 visual model to analyze the screenshot and output the recognized structured element data.

[0109] The parsing module is used to take a page URL as input and use DOM parsing functions to generate DOM structure data;

[0110] The matching module is used to call the element matching function to match visual elements and DOM structure, and to get the data of all successfully matched elements. Elements that fail to match are purely visual elements.

[0111] The element extraction module is used to extract semantic positioning structure or hybrid positioning structure data from all successfully matched elements according to the interface element extraction strategy.

[0112] The element application module is used to apply the generated structured data to automated test scripts and run the scripts to test the successful execution of the automated scripts.

[0113] The visual positioning module in this embodiment includes:

[0114] The feature extraction submodule is used to build an agent of the DeepSeek-VL2 vision model. It utilizes the powerful visual understanding capabilities of the DeepSeek-VL2 model to extract key visual features from the image and output structured results.

[0115] The conversion submodule is used to open the test page using the browser automation tool Playwright, obtain a screenshot of the test page rendering, and convert the screenshot file into a Base64 encoded string, which is then stored in the BrowserState.screenshot property;

[0116] The request body creation submodule is used to create request bodies that conform to the requirements of the visual model API;

[0117] The parsing and generation submodule is used to call the agent of the DeepSeek-VL2 vision model using a POST request, and parse the structured results output by the agent to generate structured element data in JSON format.

[0118] The parsing module in this embodiment includes:

[0119] The DOM tree basic modeling submodule defines a parent class DOMBaseNode and two subclasses DOMTextNode. The parent class DOMBaseNode is the base class for all DOM nodes, containing properties indicating whether a node is visible (is_visible) and its parent node (parent). The subclass DOMTextNode inherits from DOMBaseNode and represents text nodes. Text node information includes text content (text) and node type (type, fixed at 'TEXT_NODE'), and a method to check if there is a highlighted index in the parent node chain (has_parent_with_highlight_index). The subclass DOMElementNode inherits from DOMBaseNode and represents element nodes. Element node information includes tag name (tag_name), XPath path (xpath), attribute dictionary (attributes), child node list (children) property, and indicators of interactivity and whether it is a top-level element, as well as coordinate information. Element node information also includes a method for generating a string representation (__repr__) and a method for calculating a hash value (hash).

[0120] The function reconstruction submodule is used to reconstruct the page information corresponding to the URL through the DOM tree, converting it into DOM structure data. The DOM tree reconstruction function initializes three parameters: `doHighlightElements`, `focusHighlightIndex`, and `focusHighlightIndex`, adding an element highlighting mechanism to mark important elements. It also defines several auxiliary functions (such as XPath path generation, element visibility determination, and element interactivity detection) to parse element types, element levels, attributes, and event binding relationships. The DOM tree reconstruction function also handles special content (Shadow DOM, iframe) other than ordinary elements. `doHighlightElements` indicates whether element highlighting is enabled; `focusHighlightIndex` represents the index of the highlighted element to be focused; and `viewportExpansion` represents the viewport expansion range value.

[0121] The structured output submodule is used to perform structured output after the DOM tree is parsed. The structured output includes tagName, attributes, xpath, and attributes of children elements.

[0122] The matching module in this embodiment includes:

[0123] The key feature extraction submodule is used to process visual elements, convert visually recognized elements into standardized element objects, extract key features and perform the conversion, add a unique identifier and generate an element hash signature after the conversion is completed; among them, key features include element viewport size and scroll state, precise spatial coordinate information and advanced CSS selectors;

[0124] The hash signature generation submodule is used to process DOM nodes, add a unique identifier to the DOM node and generate an element hash signature;

[0125] The hash comparison submodule is used to recursively traverse all child nodes in the DOM structure, starting from the root node, and perform hash comparisons.

[0126] If the core attributes are consistent, the spatial positions overlap, and the structural paths are similar, then the match is considered successful, and the DOM node object is marked as an element that can be automatically manipulated.

[0127] If any one of the three requirements—identical core attributes, overlapping spatial locations, and similar structural paths—is not met, the match is considered to have failed, the DOM node object is marked as a pure visual element, and the process continues to traverse the next node.

[0128] The element extraction module in this embodiment adopts a two-level path generation strategy, providing two types of interface element extraction methods. Testers can flexibly choose according to the actual situation of the project to extract elements that can be automated, as follows:

[0129] Level 1 is semantic positioning, which is suitable for scenarios where the function of an element is stable and unchanged. The generated example is represented as: {role:"shopping cart icon"};

[0130] Level 2 is a hybrid positioning, suitable for scenarios where element text is fine-tuned. The generated example is: {text:"Settlement",xpath:" / / div[@class='footer'] / button"}.

[0131] Example 3:

[0132] This embodiment also provides an electronic device, including: a memory and a processor;

[0133] The memory stores the instructions executed by the computer.

[0134] The processor executes computer execution instructions stored in the memory, causing the processor to perform the Web interface element recognition method in any embodiment of the present invention.

[0135] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0136] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0137] Example 4:

[0138] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the Web interface element recognition method in any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0139] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0140] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0141] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0142] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing web interface elements, characterized in that, The method is as follows: Visual localization: Input the page URL address, take a screenshot using the Playwright screenshot function, convert the screenshot to a Base64 encoded string using a function, and input the string and URL into the agent. The agent calls the DeepSeek-VL2 visual model to analyze the screenshot and output the identified structured element data. DOM structure parsing: Input the page URL address, use the DOM parsing function to generate DOM structure data; DOM tree parsing and visual element matching: Call the element matching function to match visual elements and DOM structure, obtain the data of all successfully matched elements, and the elements that fail to match are pure visual elements; Interface element extraction and location generation: Based on the interface element extraction strategy, obtain semantic location structure or hybrid location structure data for all successfully matched elements; Element application: Apply the generated structured data to the automated test script, and run the script to test the successful execution of the automated script.

2. The Web interface element recognition method according to claim 1, characterized in that, The visual positioning is as follows: Construct an agent based on a DeepSeek-VL2 vision model, leveraging the powerful visual understanding capabilities of the DeepSeek-VL2 model to extract key visual features from images and output structured results; Use the browser automation tool Playwright to open the test page, capture a screenshot of the test page rendering, convert the screenshot file into a Base64 encoded string, and store it in the BrowserState.screenshot property; Create a request body that conforms to the requirements of the visual model API; Use a POST request to invoke the agent of the DeepSeek-VL2 vision model, parse the structured results output by the agent, and generate structured element data in JSON format.

3. The Web interface element recognition method according to claim 1, characterized in that, The DOM structure parsing is as follows: Complete the basic DOM tree modeling: Define a parent class DOMBaseNode and two subclasses DOMTextNode. The parent class DOMBaseNode is the base class for all DOM nodes, containing information such as node visibility and parent node properties. The subclass DOMTextNode inherits from the parent class DOMBaseNode and represents text nodes. The information of a text node includes its text content and node type, and it has a method to check if there is a highlighted index in the parent node chain. The subclass DOMElementNode inherits from the parent class DOMBaseNode and represents element nodes. The information of an element node includes its tag name, XPath path, attribute dictionary, child node list properties, and indicators of whether it is interactive and whether it is a top-level element, as well as its coordinate information. The element node information also includes a method for generating a string representation and a method for calculating a hash value. The DOM tree reconstruction function converts the page information corresponding to the URL into DOM structure data. Within this function, three parameters are initialized: `doHighlightElements`, `focusHighlightIndex`, and `focusHighlightIndex`. An element highlighting mechanism is added to mark important elements. Several helper functions are also defined to parse element types, element levels, attributes, and event bindings. The function also handles special content beyond ordinary elements. Specifically, `doHighlightElements` indicates whether element highlighting is enabled; `focusHighlightIndex` represents the index of the highlighted element to be focused; and `viewportExpansion` represents the viewport expansion range value. After parsing the DOM tree, a structured output is generated, which includes tagName, attributes, xpath, and attributes of the children element.

4. The Web interface element recognition method according to claim 1, characterized in that, The DOM tree parsing and visual element matching are as follows: The visual elements are processed by converting them into standardized element objects, extracting key features and performing the transformation. After the transformation is complete, a unique identifier is added and an element hash signature is generated. The key features include the element viewport size and scrolling status, precise spatial coordinate information, and advanced CSS selectors. Process DOM nodes by adding unique identifiers and generating element hash signatures; In the DOM structure, recursively traverse all child nodes starting from the root node and perform hash comparisons: If the core attributes are consistent, the spatial positions overlap, and the structural paths are similar, then the match is considered successful, and the DOM node object is marked as an element that can be automatically manipulated. If any one of the three requirements—identical core attributes, overlapping spatial locations, and similar structural paths—is not met, the match is considered to have failed, the DOM node object is marked as a pure visual element, and the process continues to traverse the next node.

5. The Web interface element recognition method according to any one of claims 1-4, characterized in that, The extraction and positioning of interface elements are detailed as follows: After element matching is completed, a two-level path generation strategy is adopted, providing two types of interface element extraction methods. Testers can flexibly choose according to the actual situation of the project to extract elements that can be automated, as follows: Level 1 is semantic positioning, which is suitable for scenarios where the function of an element is stable and unchanged. The generated example is represented as: {role:"shopping cart icon"}; Level 2 is a hybrid positioning, suitable for scenarios where element text is fine-tuned. The generated example is: {text:"Settlement",xpath:" / / div[@class='footer'] / button"}.

6. A web interface element recognition system, characterized in that, The system includes: The visual positioning module is used to input the page URL address, take a screenshot using the Playwright screenshot function, convert the screenshot to a Base64 encoded string using a function, and input the string and URL into the agent. The agent calls the DeepSeek-VL2 visual model to analyze the screenshot and output the recognized structured element data. The parsing module is used to take a page URL as input and use DOM parsing functions to generate DOM structure data; The matching module is used to call the element matching function to match visual elements and DOM structure, and to get the data of all successfully matched elements. Elements that fail to match are purely visual elements. The element extraction module is used to extract semantic positioning structure or hybrid positioning structure data from all successfully matched elements according to the interface element extraction strategy. The element application module is used to apply the generated structured data to automated test scripts and run the scripts to test the successful execution of the automated scripts.

7. The Web interface element recognition system according to claim 6, characterized in that, The visual positioning module includes: The feature extraction submodule is used to build an agent of the DeepSeek-VL2 vision model. It utilizes the powerful visual understanding capabilities of the DeepSeek-VL2 model to extract key visual features from the image and output structured results. The conversion submodule is used to open the test page using the browser automation tool Playwright, obtain a screenshot of the test page rendering, and convert the screenshot file into a Base64 encoded string, which is then stored in the BrowserState.screenshot property; The request body creation submodule is used to create request bodies that conform to the requirements of the visual model API; The parsing and generation submodule is used to call the agent of the DeepSeek-VL2 vision model using a POST request, and parse the structured results output by the agent to generate structured element data in JSON format; The parsing module includes: The DOM tree basic modeling submodule defines a parent class DOMBaseNode and two subclasses DOMTextNode. The parent class DOMBaseNode is the base class for all DOM nodes, containing information about node visibility and parent node properties. The subclass DOMTextNode inherits from DOMBaseNode and represents text nodes. Text node information includes text content and node type, and includes a method to check if there is a highlighted index in the parent node chain. The subclass DOMElementNode inherits from DOMBaseNode and represents element nodes. Element node information includes tag name, XPath path, attribute dictionary, child node list, and indicators of interactivity and whether it is a top-level element, as well as coordinate information. Element node information also includes a method for generating a string representation and a method for calculating a hash value. The function reconstruction submodule is used to reconstruct the page information corresponding to the URL through the DOM tree, converting it into DOM structure data. The DOM tree reconstruction function initializes three parameters: `doHighlightElements`, `focusHighlightIndex`, and `focusHighlightIndex`, adding an element highlighting mechanism to mark important elements. It also defines several helper functions to parse element types, element levels, attributes, and event binding relationships. The DOM tree reconstruction function also handles special content other than ordinary elements. Specifically, `doHighlightElements` indicates whether element highlighting is enabled; `focusHighlightIndex` represents the index of the highlighted element to be focused; and `viewportExpansion` represents the viewport expansion range value. The structured output submodule is used to perform structured output after the DOM tree is parsed. The structured output includes tagName, attributes, xpath, and attributes of children elements.

8. The Web interface element recognition system according to claim 6 or 7, characterized in that, The matching module includes: The key feature extraction submodule is used to process visual elements, convert visually recognized elements into standardized element objects, extract key features and perform the conversion, add a unique identifier and generate an element hash signature after the conversion is completed; among them, key features include element viewport size and scroll state, precise spatial coordinate information and advanced CSS selectors; The hash signature generation submodule is used to process DOM nodes, add a unique identifier to the DOM node and generate an element hash signature; The hash comparison submodule is used to recursively traverse all child nodes in the DOM structure, starting from the root node, and perform hash comparisons. If the core attributes are consistent, the spatial positions overlap, and the structural paths are similar, then the match is considered successful, and the DOM node object is marked as an element that can be automatically manipulated. If any one of the three requirements—identical core attributes, overlapping spatial locations, and similar structural paths—is not met, the match is considered to have failed, the DOM node object is marked as a pure visual element, and the process continues to traverse the next node. The element extraction module employs a two-level path generation strategy, providing two methods for extracting UI elements. Testers can flexibly choose the method based on the actual project situation to extract elements that can be automated, as detailed below: Level 1 is semantic positioning, which is suitable for scenarios where the function of an element is stable and unchanged. The generated example is represented as: {role:"shopping cart icon"}; Level 2 is a hybrid positioning, suitable for scenarios where element text is fine-tuned. The generated example is: {text:"Settlement",xpath:" / / div[@class='footer'] / button"}.

9. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the Web interface element recognition method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the Web interface element recognition method as described in any one of claims 1 to 5.

Images