Website element identification method and device based on semantic structuring

Abstract

The invention discloses a semantic structuring-based website element identification method and device. The method comprises the following steps of: obtaining document structure information of a to-be-tested website; based on the document object model structure of each webpage, generating element semantic structured information of each webpage; reasoning the semantic structured information of each element by using a large language model to obtain an element positioning process of a plurality of webpage elements; and generating an execution script of the automatic test tool based on the element positioning process of each webpage element. According to the method, a large language model is utilized, semantic structured information of each element is combined, an element positioning process of a plurality of webpage elements is obtained, manpower cost does not need to be spent to participate in data annotation, an execution script is generated based on the element positioning process, the problem of webpage element automatic interaction failure during website content change can be solved, and the user experience is improved. Therefore, effective and reliable webpage element identification is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of website automatic interaction, and in particular to a website element identification method and device based on semantic structuring. BACKGROUND

[0002] Current website automatic interaction technology mainly relies on DOM (Document Object Model) selectors (such as CSS selectors), visual recognition technology and reinforcement learning methods for web element identification. These methods have their own limitations (specifically, DOM selectors are highly fragile and extremely sensitive to structural changes, lacking semantic reasoning; visual recognition is computationally intensive, slow and greatly affected by visual changes; reinforcement learning has high training costs, slow convergence and requires a large number of samples). These existing methods are extremely sensitive to website structural changes, and minor changes in website front-end code can cause the methods to fail. SUMMARY

[0003] The present application provides a website element identification method and device based on semantic structuring, aiming to achieve effective and reliable web element identification.

[0004] To achieve the above purpose, the present application provides the following technical solutions:

[0005] A website element identification method based on semantic structuring, comprising:

[0006] Obtaining document structure information of a website to be tested; the document structure information includes document object model structures of multiple web pages;

[0007] Based on the document object model structure of each web page, generating element semantic structuring information of each web page;

[0008] Using a large language model to reason on each element semantic structuring information to obtain element positioning procedures of multiple web elements; the element positioning procedure includes a flow description text for positioning the web element;

[0009] Based on the element positioning procedure of each web element, generating an execution script of an automatic test tool; the execution script is used to implement the identification operation of each web element.

[0010] Optionally, based on the document object model structure of each web page, generating element semantic structuring information of each web page, comprises:

[0011] For each web page, information extraction is performed on the document object model structure of the web page to obtain multiple web elements and corresponding attribute information; the attribute information at least includes label, element ID, class name, text content and state;

[0012] determine the type of each of the webpage elements based on the tag and class name of each of the webpage elements;

[0013] determine the function of each of the webpage elements based on the text content of each of the webpage elements;

[0014] perform context analysis on the document object model structure of each of the webpages to obtain the positional relationship and parent container of each of the webpage elements;

[0015] invoke a target expert knowledge base to perform intent analysis on the attribute information of each of the webpage elements to obtain the user intent of each of the webpage elements;

[0016] load the element ID, type, function, state, text content, positional relationship, parent container, and user intent of each of the webpage elements into a preset template to generate element semantic structured information of each of the webpages.

[0017] Optionally, the loading of the element ID, type, function, state, text content, positional relationship, parent container, and user intent of each of the webpage elements into the preset template to generate the element semantic structured information of each of the webpages comprises:

[0018] perform visual feature analysis on each of the webpages to obtain the container color of each of the webpage elements;

[0019] construct a semantic network based on the type and function of each of the webpage elements; the semantic network comprises the association relationship between the webpage elements;

[0020] analyze the association relationship between the webpage elements to obtain the related elements and interaction consequences of each of the webpage elements;

[0021] configure the priority of each of the webpage elements;

[0022] load the element ID, type, function, state, text content, positional relationship, parent container, container color, related elements, user intent, interaction consequences, and priority of each of the webpage elements into the preset template to generate the element semantic structured information of each of the webpages.

[0023] Optionally, a large language model is used to perform reasoning on the element semantic structured information to obtain element positioning procedures of multiple webpage elements, which comprises:

[0024] determine the corresponding system prompt word based on the user input query intent; the query intent comprises a question description text for querying webpage elements;

[0025] Based on the system prompt words and the semantic structured information of each element, as input to a large language model, an inference result output by the large language model is obtained; the inference result includes an element positioning process of multiple web page elements.

[0026] Optionally, based on the element location process of each web page element, an execution script of an automated testing tool is generated, including:

[0027] For each web page element, using a natural language processing model, converting the element location process of the web page element into an operation intention that can be recognized by an automated testing tool; the operation intention includes at least a plurality of operation strategies related to the web page element;

[0028] Generate a selector corresponding to the plurality of operation strategies; the selector is used to search for the web page element in the web page;

[0029] Determining the compatibility between the web page element and each of the selectors based on a designated selector evaluation engine;

[0030] Generate the corresponding command statement based on the target selector with the highest adaptability;

[0031] Based on the command statement corresponding to each of the web page elements, an execution script of the automated testing tool is generated.

[0032] Optionally, generating an execution script of the automated testing tool based on the command statement corresponding to each web page element includes:

[0033] Sorting the selectors in descending order of fitness to obtain a corresponding selector sequence;

[0034] Based on the selector sequence, a corresponding selector manager is generated; the selector manager is used to call other selectors in the selector sequence to generate and execute new command statements when the target selector fails to execute;

[0035] Based on the command statement and selector manager corresponding to each of the web page elements, an execution script of the automated testing tool is generated.

[0036] Optionally, the method further includes:

[0037] In response to a test instruction triggered by a user, control the automated testing tool to run the execution script to obtain corresponding test results;

[0038] Based on the test results, determining a valid selector that was successfully executed;

[0039] According to the effective selector, the execution script is optimized to obtain an optimized execution script.

[0040] Optionally, the method further comprises:

[0041] Based on the test result, an effective element positioning process that is successfully executed is determined, and a total number of the effective element positioning processes is counted.

[0042] If the total number does not meet a preset number requirement, a new system prompt word is determined based on the effective element positioning process.

[0043] The new system prompt word is input into the large language model, and each element semantic structured information is inferred again using the large language model to obtain new element positioning processes of the plurality of web page elements.

[0044] Based on the new element positioning process of each web page element, a new execution script is generated.

[0045] A website element recognition device based on semantic structuring, comprising:

[0046] A document structure acquisition unit is configured to acquire document structure information of a website to be tested; the document structure information comprises document object model structures of a plurality of web pages.

[0047] A semantic structure generation unit is configured to generate element semantic structured information of each web page based on the document object model structure of each web page.

[0048] A positioning process inference unit is configured to infer each element semantic structured information using a large language model to obtain element positioning processes of a plurality of web page elements; the element positioning processes comprise process description texts for positioning the web page elements.

[0049] An execution script generation unit is configured to generate an execution script of an automated test tool based on the element positioning process of each web page element; the execution script is used to implement a recognition operation of each web page element.

[0050] A storage medium comprises a stored program, wherein the program is executed by a processor to perform the website element recognition method based on semantic structuring.

[0051] An electronic device comprises a processor, a memory and a bus; the processor is connected with the memory through the bus;

[0052] The memory is configured to store a program, and the processor is configured to execute the program, wherein the program is executed by the processor to perform the website element recognition method based on semantic structuring.

[0053] The technical solution provided in the application obtains document structure information of a website to be tested, generates element semantic structured information of each web page based on a document object model structure of each web page, uses a large language model to infer the element semantic structured information of each element to obtain an element positioning process of multiple web page elements, and generates an execution script of an automated test tool based on the element positioning process of each web page element. The application uses a large language model to obtain the element positioning process of multiple web page elements in combination with element semantic structured information, does not need to spend labor cost for data labeling, and can solve the problem of invalid web page element automatic interaction when website content is changed, thereby realizing effective and reliable web page element identification. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0055] Figure 1 A flowchart of a website element identification method based on semantic structuring provided by an embodiment of the present application;

[0056] Figure 2 A flowchart of another website element identification method based on semantic structuring provided by an embodiment of the present application;

[0057] Figure 3 A flowchart of another website element identification method based on semantic structuring provided by an embodiment of the present application;

[0058] Figure 4 A flowchart of another website element identification method based on semantic structuring provided by an embodiment of the present application;

[0059] Figure 5 A flowchart of another website element identification method based on semantic structuring provided by an embodiment of the present application;

[0060] Figure 6 A flowchart of another website element identification method based on semantic structuring provided by an embodiment of the present application;

[0061] Figure 7 An architecture diagram of a website element identification device based on semantic structuring provided by an embodiment of the present application;

[0062] Figure 8 A schematic diagram of element semantic structured information provided in an embodiment of the present application;

[0063] Figure 9 A logical diagram of a script execution provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0065] In this application, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. The terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or apparatus comprising the element.

[0066] like Figure 1 FIG. 1 is a flow chart of a website element identification method based on semantic structuring provided in an embodiment of the present application, which includes the following steps.

[0067] S101: Obtain document structure information of the website to be tested.

[0068] The document structure information includes the document object model structures of multiple web pages.

[0069] In some examples, the Document Object Model structure may include a DOM tree. DOM is a representation format for identifying HTML and XML documents. DOM maps the structure of a web page to programmable objects. Specifically, scripting languages ​​such as DOM and JavaScript can conveniently access, modify, and manipulate the content and structure of a web page. The DOM tree is a visual representation of the document structure. Each part of the document is considered a node. The root node of the DOM tree is usually the root element of the document, such as the tag of an HTML document. The DOM tree contains various elements in the HTML document from top to bottom, and each element node may contain child nodes.

[0070] In a possible implementation, the types of nodes in a DOM tree can be: element nodes, text nodes, attribute nodes, and document nodes. Specifically, element nodes correspond to HTML tags (e.g. 、 The text node represents the text content in the element, the attribute node represents the attribute (for example, class, id, etc.) of the HTML element, and the document node represents the root node of the entire HTML document (usually an element).

[0071] It should be noted that the byte stream of the HTML document of the plurality of web pages in the website can be obtained directly from the server of the website to be tested through the browser, and the byte stream is converted into a character stream according to the encoding mode (for example, UTF-8, UTF-16, etc.) of the HTML document, and then a lexical analyzer is called to perform lexical analysis on the character stream to obtain a token sequence, and then a syntax analyzer is called to perform syntax analysis on the token sequence to obtain the DOM tree of each web page.

[0072] In possible embodiments, the lexical analyzer can scan the character stream in a specified order, and decompose the character stream into individual tokens.

[0073] In possible embodiments, the syntax analyzer determines the relationship between the tokens according to the syntax rules of HTML, combines related tokens into nodes, and constructs the parent-child relationship between the nodes.

[0074] S102: Based on the document object model structure of each web page, generate element semantic structured information of each web page.

[0075] After obtaining the DOM tree of each web page, the corresponding feature information can be extracted from the DOM tree of each web page to generate the element semantic structured information of each web page. The element semantic structured information includes semantic structure text of a plurality of web page elements, which can accurately express the function, position and relationship of the web page elements.

[0076] Optionally, the implementation process of generating element semantic structured information of each web page based on the document object model structure of each web page can be referred to Figure 2 The steps shown and the corresponding explanations.

[0077] S103: Using a large language model, reasoning on each element semantic structured information to obtain an element positioning process of a plurality of web page elements.

[0078] The element positioning process includes a flow description text for positioning the web page elements.

[0079] It should be noted that LLM (Large Language Model) refers to a deep learning model trained using a large amount of text data, which enables the model to generate natural language text or understand the meaning of language text.

[0080] Optionally, the element semantic structured information of each element is inferred by using a large language model to obtain an implementation process of the element positioning process of the plurality of webpage elements, which can be: determining a corresponding system prompt based on a query intention input by a user; the query intention includes a question description text for querying the webpage element; taking the system prompt and the element semantic structured information of each element as input of the large language model to obtain an inference result output by the large language model; the inference result includes the element positioning process of the plurality of webpage elements.

[0081] In some examples, the system prompt refers to an initial instruction or background information set for a large language model (such as GPT, LLaMA, etc.) for guiding the behavior and response mode of the large language model.

[0082] In possible implementations, the query intention input by the user can be: finding an "immediate purchase" button on the page.

[0083] In some examples, taking the webpage element corresponding to the "immediate purchase" button in the product purchase page of the shopping website as an example, the element positioning process output by the large language model can be seen from steps 1-7.

[0084] Step 1, input processing: user query intention, "find the purchase button on the page"; load element semantic structured information of each webpage.

[0085] Step 2, query understanding: core intention recognition, find (action) purchase button (target object); query expansion: "purchase" -> "purchase / immediate purchase / place an order / checkout".

[0086] Step 3, candidate element screening: initially screen all button type elements (15); apply function relevance filtering to retain purchase related buttons (3); calculate similarity scores and sort candidate elements.

[0087] Step 4, element confirmation: highest confidence match: "immediate purchase" button (confidence 92%); second highest confidence match: "add to cart" button (confidence 65%); confirm that the best match is the "immediate purchase" button.

[0088] Step 5, selector strategy generation: generate ID selector: #btn-buy (robustness: high, simplicity: high); generate text selector: button:contains("immediate purchase") (robustness: medium, simplicity: medium); generate hierarchical selector: product-detail.action-area button.primary-btn (robustness: medium-high, simplicity: low).

[0089] Step 6, selector strategy ranking: optimal selector: #btn-buy; alternative selector: button: contains ("Buy Now"); final recommended strategy combination: try ID selector, and use text selector if failed.

[0090] Step 7, execute feedback: selector successfully locates element; update target expert knowledge base to enhance understanding of elements in the website to be tested.

[0091] It can be understood that the content shown in steps 1-7 above is only for illustration.

[0092] S104: Based on the element positioning process of each web element, an execution script of the automated testing tool is generated.

[0093] The execution script is used to implement the recognition operation of each web element.

[0094] Optionally, the implementation process of generating the execution script of the automated testing tool based on the element positioning process of each web element can be referred to the steps shown in Figure 3 and the corresponding explanation.

[0095] The processes shown in S101-S104 above use a large language model in combination with various element semantic structured information to obtain the element positioning process of multiple web elements, without the need for human labor to participate in data labeling. Moreover, the execution script generated based on the element positioning process can solve the problem of web element automation interaction failure when the website content changes, thereby realizing effective and reliable web element recognition.

[0096] As shown in Figure 2 , a process diagram of another website element recognition method based on semantic structuring provided by the embodiments of the present application is shown, which includes the following steps.

[0097] S201: For each web page, information extraction is performed on the document object model structure of the web page to obtain multiple web elements and corresponding attribute information.

[0098] The attribute information at least includes tags, element IDs, class names, text content, and states.

[0099] In possible embodiments, taking the web element corresponding to the "Buy Now" button in the product purchase page of a shopping website as an example, the expression form of the obtained web element can be: <button id="btn-buy" class="primary-btn action-buy">Buy now< / button> .

[0100] In possible implementation, taking the webpage element corresponding to the "Buy Now" button in the commodity purchase page of the shopping website as an example, the attribute information of the obtained webpage element can be: label (button), ID (btn-buy), class name (primary-btn, action-buy), text content ("Buy Now"), and state (clickable).

[0101] S202: Determine the type of each webpage element based on the label and class name of each webpage element.

[0102] The type of the webpage element can be determined by performing semantic analysis on the label and class name of the webpage element.

[0103] In possible implementation, taking the webpage element corresponding to the "Buy Now" button in the commodity purchase page of the shopping website as an example, the type of the "Buy Now" button can be determined as an operation button by performing semantic analysis on the label (button) and class name (primary-btn, action-buy) of the "Buy Now" button.

[0104] S203: Determine the function of each webpage element based on the text content of each webpage element.

[0105] The function of the webpage element can be determined by performing keyword recognition on the text content of the webpage element.

[0106] In possible implementation, taking the webpage element corresponding to the "Buy Now" button in the commodity purchase page of the shopping website as an example, the function of the "Buy Now" button can be determined as a commodity purchase function by performing keyword recognition on the text content ("Buy Now") of the "Buy Now" button.

[0107] S204: Perform context analysis on the document object model structure of each webpage to obtain the positional relationship and parent container of each webpage element.

[0108] The positional relationship and parent container of each webpage element can be obtained by performing context analysis on the document object model structure of each webpage using a pre-trained language model (such as BERT model).

[0109] In possible implementation, taking the commodity purchase page of the shopping website as an example, the positional relationship (such as located at the bottom of the commodity detail page), parent container (such as located in the operation area container of the commodity detail page), and sibling element (such as having a "Add to Cart" button next to it) of the "Buy Now" button can be obtained by performing context analysis on the DOM tree of the commodity purchase page.

[0110] S205: performing visual feature analysis on each webpage to obtain the container color of each webpage element.

[0111] The visual feature analysis mainly includes extracting and analyzing various visual features of each webpage element in the webpage, including color, shape, texture, motion, etc.

[0112] In possible embodiments, taking the webpage element corresponding to the "Buy Now" button in the product purchase page of a shopping website as an example, through visual feature analysis on the product purchase page, the container color (e.g., yellow background) of the "Buy Now" button, the font style (e.g., black text and 16px font), the position information (e.g., located at the bottom of the page, 120px wide and 40px high), and the visibility (e.g., visible, not blocked, and within the viewport) can be obtained.

[0113] S206: constructing a semantic network based on the type and function of each webpage element.

[0114] The semantic network includes the association relationship between each webpage element.

[0115] In some examples, the SN (Semantic Network) method can be used to formalize the type and function of each webpage element to generate the corresponding semantic network. Generally, the SN method is a formalization analysis method based on the semantic network knowledge representation method, which models the type, function, and other elements through nodes and relationship chains to formally express the association relationship between each webpage element.

[0116] S207: analyzing the association relationship between each webpage element to obtain the related elements and the interaction consequences of each webpage element.

[0117] By analyzing the association relationship between each webpage element, the related elements and the workflow of each webpage element can be obtained, and the interaction results of the webpage element can be further obtained by analyzing the workflow.

[0118] In possible embodiments, taking the webpage element corresponding to the "Buy Now" button in the product purchase page of a shopping website as an example, by analyzing the association relationship between each webpage element in the product purchase page, the related elements (e.g., product information, price label, shopping cart icon, and payment system) and the workflow (e.g., viewing product → selecting price → buying immediately → settling payment) of the "Buy Now" button can be determined, and the interaction results (e.g., jumping to the order confirmation page) of the "Buy Now" button can be determined by analyzing the workflow.

[0119] S208: Calling the target expert knowledge base to perform intent analysis on the attribute information of each web page element to obtain the user intent of each web page element.

[0120] The target expert knowledge base can be used to provide professional knowledge that matches the website to be tested. The specific target expert knowledge base can be set by technical personnel based on actual conditions. In addition, the user intent of a web page element is used to represent the service effect that can be achieved after the web page element is triggered by the user.

[0121] In a possible implementation, taking the web page element corresponding to the "Buy Now" button on the product purchase page of a shopping website as an example, the e-commerce expert knowledge base is called to perform intent analysis on the attribute information of the "Buy Now" button to obtain the user intention of the "Buy Now" button (for example, quickly completing the purchase process).

[0122] S209: Configure the priority of each web page element.

[0123] The priority of each web page element may be configured according to a preset priority matching rule and in combination with the type of each web page element.

[0124] In a possible implementation manner, the preset priority matching rule includes element types matched at multiple priority levels.

[0125] S210: Loading the element ID, type, function, status, text content, position relationship, parent container, container color, related elements, user intention, interaction consequences and priority of each web page element into a preset template to generate element semantic structured information of each web page.

[0126] Among them, based on the element ID, type, function, status, text content, position relationship, parent container, container color, related elements, user intention, interaction consequences and priority of the web page elements, as element semantic structured information, the DOM tree of the web page can be converted into structured text that can be recognized by large language models.

[0127] In some examples, taking the ordering scenario of a shopping website as an example, the semantic structured information of the elements obtained under each web page can be found in Figure 8 shown.

[0128] The above process shown in S201-S210 can effectively convert the document object model structure of each web page into the element semantic structured information of each web page.

[0129] like Figure 3 As shown, it is a flow chart of another website element identification method based on semantic structuring provided in an embodiment of the present application, which includes the following steps.

[0130] S301: For each web page element, the element positioning process of the web page element is converted into an operation intention that can be recognized by an automated test tool by using a natural language processing model.

[0131] The operation intention includes at least a plurality of operation strategies related to the web page element.

[0132] S302: A selector corresponding to the plurality of operation strategies is generated.

[0133] The selector is used to find the web page element in the web page.

[0134] In some examples, the selector can be a DOM selector, which is a method provided by a document object for finding the first element in the document that matches the specified CSS selector. The selector can be a class name (such as.active), an ID (such as #myId), a tag name (such as div), or any valid CSS selector.

[0135] S303: Determine the degree of adaptation between the web page element and each selector based on the specified selector evaluation engine.

[0136] The specified selector evaluation engine can use CSS-selcet, which is a CSS selector compiler and engine that can convert CSS selectors into functions to test whether elements match these selectors. As an engine, CSS-selcet can search for web page elements in the DOM tree and perform element matching according to the way browsers execute CSS selectors.

[0137] It can be understood that the specified selector evaluation engine is used to perform matching between the web page element and each selector, thereby obtaining the degree of adaptation between the web page element and each selector.

[0138] S304: Generate a corresponding command statement based on the target selector with the highest degree of adaptation.

[0139] Generating a corresponding command statement based on the target selector with the highest degree of adaptation can improve the success rate of website element recognition.

[0140] S305: Generate an execution script of an automated test tool based on the command statement corresponding to each web page element.

[0141] The execution script can realize the automatic interaction between the automated test tool and the website to be tested, thereby completing the automatic recognition task of each web page element in the website to be tested.

[0142] In some examples, to further increase the execution success rate of the execution script, a selector manager can be added to the execution script, which is used to call other selectors in the selector sequence in the case of target selector execution failure, generate and execute new command statements to ensure successful identification of web page elements. Optionally, based on the command statement corresponding to each web page element, the implementation process of the execution script of the automation testing tool can be generated, which can be referred to as Figure 4 the steps shown and the corresponding explanation.

[0143] In some examples, to further increase the execution success rate of the execution script, the test results of the execution script can be used to optimize and update the execution script. Optionally, the optimization and updating process of the execution script can be referred to as Figure 5 the steps shown and the corresponding explanation.

[0144] In some examples, to improve the success rate of web element identification in the website to be tested, the test results of the execution script can also be used to optimize and update the element positioning process of each web page element. Optionally, the optimization and updating process of the element positioning process can be referred to as Figure 6 the steps shown and the corresponding explanation.

[0145] In some examples, taking the element positioning process with the user query intention of "finding Nan Bowen baseball uniform entering the product detail page and placing an order" as an example, based on the methods shown in Figure 3 and Figure 4 the generated execution script can be used to implement the logic shown in Figure 9 .

[0146] The processes shown in S301-S305 can interact with various web pages with high adaptability and stability, effectively solving the problem of automation failure caused by changes in website content.

[0147] As shown in Figure 4 , another process diagram of a website element identification method based on semantic structuring provided by an embodiment of the present application is shown, which includes the following steps.

[0148] S401: Sort each selector in order of decreasing adaptability to obtain a corresponding selector sequence.

[0149] In which, sorting each selector in order of decreasing adaptability can facilitate the selector manager to select selectors with high adaptability.

[0150] S402: Based on the selector sequence, generate a corresponding selector manager.

[0151] The selector manager is configured to invoke other selectors in the selector sequence to generate and execute new command statements in the case of a target selector execution failure.

[0152] In some examples, the selector manager is specifically configured to sequentially invoke the selectors in the selector sequence according to the order from front to back to generate and execute new command statements until the selector can successfully match the web page element.

[0153] S403: Based on the command statement corresponding to each web page element and the selector manager, an execution script of the automated testing tool is generated.

[0154] The execution script containing the selector management can significantly increase the success rate of web page element recognition.

[0155] It can be understood that when the website content changes, the attribute information of the web page element changes, and the selector management is used to continuously invoke the selectors in the selector sequence to generate and execute new command statements until the selector can successfully match the web page element, thereby effectively solving the problem of automation failure caused by website content changes.

[0156] The above-mentioned S401-S403 shows the flow, and the selector manager is added in the execution script, which can improve the success rate of web page element recognition, thereby effectively solving the problem of automation failure caused by website content changes.

[0157] As shown in Figure 5 , another flowchart of a website element recognition method based on semantic structuring provided by an embodiment of the present application is shown, which includes the following steps.

[0158] S501: In response to a test instruction triggered by a user, an automated testing tool is controlled to run an execution script to obtain a corresponding test result.

[0159] The test result includes the recognition result of each web page element in the website to be tested. If the recognition result of the web page element is successful, the state of the selector matched by the web page element is execution success. If the recognition result of the web page element is failure, the state of the selector matched by the web page element is execution failure.

[0160] S502: Based on the test result, a valid selector with execution success is determined.

[0161] Based on the web page element with a successful recognition result shown in the test result, the valid selector with execution success can be determined.

[0162] S503: According to the valid selector, the execution script is optimized to obtain an optimized execution script.

[0163] In the optimized execution script, the selector sequence matched by the selector manager changes. Specifically, the order of each selector in the selector sequence is readjusted by updating the fitness of each selector in the selector sequence.

[0164] In some examples, valid selectors in the selector sequence may be retained, along with other selectors that have not been called, while other selectors that have been called and failed to execute may be removed.

[0165] The process shown in S501-S503 above can utilize the test results of the execution script to optimize and update the execution script, thereby improving the stability and reliability of the execution script.

[0166] like Figure 6 FIG. 1 is a flow chart of another method for identifying website elements based on semantic structuring provided in an embodiment of the present application, which includes the following steps.

[0167] S601: Based on the test result, determine the valid element location processes that have been successfully executed, and count the total number of valid element location processes.

[0168] Among them, the test results include the recognition results of each web page element. If the recognition result of the web page element is successful, the status of the element positioning process corresponding to the web page element is execution success. If the recognition result of the web page element is failure, the status of the element positioning process corresponding to the web page element is execution failure.

[0169] S602: If the total quantity does not meet the preset quantity requirement, it is determined as a new system prompt word based on the valid element positioning process.

[0170] Among them, if the total number of valid element positioning processes does not meet the preset number requirements, it can be determined that the reasoning ability of the large language model is insufficient. Therefore, it is necessary to reuse the new system prompt words to improve the reasoning ability of the large language model to ensure the output of a more reliable element positioning process.

[0171] S603: Input the new system prompt word into the large language model, and use the large language model again to infer the semantic structured information of each element to obtain a new element positioning process for multiple web page elements.

[0172] Among them, inputting the new system prompt words into the large language model can improve the reasoning ability of the large language model, so that the large language model can be used again to reason about the semantic structured information of each element, and a more reliable new element positioning process can be obtained.

[0173] S604: Generate a new execution script based on the new element positioning process of each web page element.

[0174] The new element positioning process based on each web page element generates a new execution script, and the success rate of the new execution script in identifying the web page element is improved, thereby effectively solving the problem of automation failure caused by changes in website content.

[0175] The processes S601-S604 can use the test results of the execution script to improve the inference ability of the large language model, optimize the update element positioning process, and thereby improve the success rate of identifying the web page element.

[0176] As shown in Figure 7 The architecture of the website element identification device based on semantic structuring provided by the embodiment of the application is shown in the following schematic diagram, which includes the following units.

[0177] The document structure acquisition unit 100 is configured to acquire document structure information of a website to be tested. The document structure information includes document object model structures of multiple web pages.

[0178] The semantic structure generation unit 200 is configured to generate element semantic structuring information of each web page based on the document object model structure of each web page.

[0179] Optionally, the semantic structure generation unit 200 is specifically configured to: for each web page, perform information extraction on the document object model structure of the web page to obtain multiple web page elements and corresponding attribute information; the attribute information at least includes a label, an element ID, a class name, text content, and a state; determine the type of each web page element based on the label and the class name of each web page element; determine the function of each web page element based on the text content of each web page element; perform context analysis on the document object model structure of each web page to obtain the position relationship and the parent container of each web page element; call a target expert knowledge base to perform intent analysis on the attribute information of each web page element to obtain the user intent of each web page element; and load the element ID, the type, the function, the state, the text content, the position relationship, the parent container, and the user intent of each web page element into a preset template to generate the element semantic structuring information of each web page.

[0180] Optionally, the semantic structure generation unit 200 is specifically configured to: perform visual feature analysis on each web page to obtain the container color of each web page element; construct a semantic network based on the type and the function of each web page element; the semantic network includes the association relationship between the web page elements; analyze the association relationship between the web page elements to obtain the related elements and the interaction consequences of each web page element; configure the priority of each web page element; and load the element ID, the type, the function, the state, the text content, the position relationship, the parent container, the container color, the related elements, the user intent, the interaction consequences, and the priority of each web page element into the preset template to generate the element semantic structuring information of each web page.

[0181] The positioning process reasoning unit 300 is configured to utilize the large language model to reason the element semantic structured information to obtain element positioning processes of the plurality of webpage elements; the element positioning process includes process description text for positioning the webpage element.

[0182] Optionally, the positioning process reasoning unit 300 is specifically configured to: determine a corresponding system prompt word based on a user input query intent; the query intent includes question description text for querying the webpage element; and input the system prompt word and the element semantic structured information into the large language model to obtain a reasoning result output by the large language model; the reasoning result includes the element positioning processes of the plurality of webpage elements.

[0183] The execution script generation unit 400 is configured to generate an execution script of an automated testing tool based on the element positioning process of each webpage element; the execution script is used to implement the identification operation of each webpage element.

[0184] Optionally, the execution script generation unit 400 is specifically configured to: generate a selector corresponding to a plurality of operation strategies; the selector is used to find the webpage element in the webpage; determine an adaptation degree between the webpage element and each selector based on a specified selector evaluation engine; generate a corresponding command statement based on a target selector with the highest adaptation degree; and generate the execution script of the automated testing tool based on the command statement corresponding to each webpage element.

[0185] Optionally, the execution script generation unit 400 is specifically configured to: sort the selectors in a descending order of the adaptation degrees to obtain a corresponding selector sequence; generate a corresponding selector manager based on the selector sequence; the selector manager is used to call other selectors in the selector sequence to generate and execute a new command statement in the case that the target selector fails to execute; and generate the execution script of the automated testing tool based on the command statement corresponding to each webpage element and the selector manager.

[0186] Optionally, the execution script generation unit 400 is further configured to: control the automated testing tool to run the execution script to obtain a corresponding test result in response to a test instruction triggered by a user; determine an effective selector that successfully executes based on the test result; and optimize the execution script according to the effective selector to obtain an optimized execution script.

[0187] Optionally, the execution script generation unit 400 is further configured to: determine an effective element positioning process that succeeds based on the test result, and count a total number of effective element positioning processes; if the total number does not meet a preset number requirement, determine a new system prompt word based on the effective element positioning process; input the new system prompt word into the large language model, and again use the large language model to reason the semantic structured information of each element to obtain new element positioning processes of the plurality of webpage elements; and generate a new execution script based on the new element positioning process of each webpage element.

[0188] The above-described various units use the large language model to obtain the element positioning processes of the plurality of webpage elements in combination with the semantic structured information of each element, without the need for human cost to participate in data labeling, and the execution script generated based on the element positioning processes can solve the problem of webpage element automatic interaction failure when website content changes, thereby realizing effective and reliable webpage element identification.

[0189] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the semantic structured website element identification method provided by the present application.

[0190] The present application also provides an electronic device, which includes a processor, a memory and a bus. The processor is connected with the memory through the bus, the memory is configured to store a program, and the processor is configured to run the program, wherein the program runs to execute the semantic structured website element identification method provided by the present application.

[0191] In addition, the functions described above in the embodiments of the present application can be at least partially performed by one or more hardware logic components. For example, non-limitingly, exemplary types of hardware logic components that can be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on a chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0192] Although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination.

[0193] The above description is only the preferred embodiment of the present application and the explanation of the technical principles. It should be pointed out that those skilled in the art should infer the disclosed range of the present application, which is not limited to the technical scheme formed by the specific combination of the technical features described above, and should also cover other technical schemes formed by the combination of the technical features described above or their equivalent features without departing from the disclosed concept. For example, the technical scheme formed by the mutual replacement of the features described above and the technical features disclosed in the present application (but not limited to) with similar functions.

Claims

1. A website element identification method based on semantic structuring, characterized in that: include: Obtaining document structure information of the website to be tested; the document structure information includes document object model structures of multiple web pages; generating element semantic structured information of each web page based on the document object model structure of each web page; Using a large language model, reasoning on the semantic structured information of each element to obtain an element location process for multiple web page elements; the element location process includes a process description text for locating the web page elements; generating an execution script of an automated testing tool based on the element location process of each web page element; The execution script is used to implement the identification operation of each web page element.

2. The method according to claim 1, characterized in that Generating element semantic structured information of each web page based on the document object model structure of each web page, including: For each web page, extract information from the document object model structure of the web page to obtain multiple web page elements and corresponding attribute information; the attribute information includes at least a tag, an element ID, a class name, text content, and a status; Determining the type of each web page element based on the tag and class name of each web page element; Determining a function of each web page element based on the text content of each web page element; Performing context analysis on the document object model structure of each web page to obtain the position relationship and parent container of each web page element; Calling a target expert knowledge base to perform intent analysis on the attribute information of each web page element to obtain the user intent of each web page element; The element ID, type, function, status, text content, position relationship, parent container, and user intention of each web page element are loaded into a preset template to generate element semantic structured information of each web page.

3. The method according to claim 2, characterized in that The element ID, type, function, status, text content, position relationship, parent container, and user intention of each web page element are loaded into a preset template to generate element semantic structured information of each web page, including: Performing visual feature analysis on each web page to obtain a container color of each web page element; Building a semantic network based on the type and function of each web page element; the semantic network includes associations between each web page element; Analyzing the association relationship between each of the web page elements to obtain the related elements and interaction consequences of each of the web page elements; Configuring the priority of each of the web page elements; The element ID, type, function, status, text content, position relationship, parent container, container color, related elements, user intention, interaction consequences and priority of each web page element are loaded into a preset template to generate element semantic structured information of each web page.

4. The method according to claim 1, wherein Using a large language model, reasoning about the semantic structured information of each element is performed to obtain the element location process of multiple web page elements, including: Determine the corresponding system prompt word based on the query intention input by the user; the query intention includes a question description text for querying web page elements; Based on the system prompt words and the semantic structured information of each element, as input to a large language model, an inference result output by the large language model is obtained; the inference result includes an element positioning process of multiple web page elements.

5. The method according to claim 1, wherein Based on the element location process of each web page element, an execution script of an automated testing tool is generated, including: For each web page element, using a natural language processing model, converting the element location process of the web page element into an operation intention that can be recognized by an automated testing tool; the operation intention includes at least a plurality of operation strategies related to the web page element; Generate a selector corresponding to the plurality of operation strategies; the selector is used to search for the web page element in the web page; Determining the compatibility between the web page element and each of the selectors based on a designated selector evaluation engine; Generate the corresponding command statement based on the target selector with the highest adaptability; Based on the command statement corresponding to each of the web page elements, an execution script of the automated testing tool is generated.

6. The method according to claim 5, characterized in that Generating an execution script of the automated testing tool based on the command statement corresponding to each web page element includes: Sort the selectors in descending order of fitness to obtain a corresponding selector sequence; Based on the selector sequence, a corresponding selector manager is generated; the selector manager is used to call other selectors in the selector sequence to generate and execute new command statements when the target selector fails to execute; Based on the command statement and selector manager corresponding to each of the web page elements, an execution script of the automated testing tool is generated.

7. The method according to claim 6, characterized in that The method further comprises: In response to a test instruction triggered by a user, control the automated testing tool to run the execution script to obtain corresponding test results; Based on the test results, determining a valid selector that was successfully executed; The execution script is optimized according to the valid selector to obtain an optimized execution script.

8. The method according to claim 7, characterized in that The method further comprises: Based on the test results, determining the successful execution of valid element location processes, and counting the total number of the valid element location processes; If the total quantity does not meet the preset quantity requirement, determining it as a new system prompt word based on the valid element positioning process; Inputting the new system prompt word into the large language model, and again using the large language model to infer the semantic structured information of each element to obtain a new element positioning process for the plurality of web page elements; Based on the new element positioning process of each web page element, a new execution script is generated.

9. A website element recognition device based on semantic structuring, characterized in that: include: A document structure acquisition unit, configured to acquire document structure information of the website to be tested; the document structure information includes a document object model structure of multiple web pages; A semantic structure generating unit, configured to generate element semantic structured information of each web page based on the document object model structure of each web page; A positioning process inference unit is used to use a large language model to infer the semantic structured information of each element to obtain an element positioning process for multiple web page elements; the element positioning process includes a process description text for locating the web page elements; An execution script generating unit, configured to generate an execution script of an automated testing tool based on an element positioning process of each web page element; The execution script is used to implement the identification operation of each web page element.

10. An electronic device, characterized in that: include: processor, memory, and bus; The processor is connected to the memory via the bus; The memory is used to store a program, and the processor is used to run the program, wherein the program, when run by the processor, executes the website element identification method based on semantic structure according to any one of claims 1 to 8.

Images