Webpage action execution method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a webpage action execution method and device, electronic equipment and a storage medium, and the method comprises the following steps: determining a function node tree based on a page function structured analysis result, and carrying out semantic matching on a task target vector corresponding to a task text and a function description vector to obtain a candidate operation path; executing a webpage action based on the candidate operation path and the matching degree sequence corresponding to the candidate operation path, when a state verification result corresponding to the webpage action is that verification is not passed, performing path backtracking on the function node tree to obtain an operation path, executing the webpage action based on the operation path, and when an execution result corresponding to the webpage action is that verification is passed, performing path backtracking on the function node tree to obtain the operation path. And displaying the execution result of the webpage action. The function node tree is determined based on the page function structured analysis result, complex hierarchical operation is effectively processed, the operation path is obtained based on semantic matching of the task target vector and the function description vector, and the accuracy of page state tracking corresponding to the operation path is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a web page action execution method, device, electronic device and storage medium. Background Art

[0002] Web automation technology has been developing continuously in recent years, but it still faces some technical limitations. Traditional web automation tools (such as Selenium and Puppeteer) mainly define operation steps and element positioning rules through manual scripting. This method has significant limitations when facing dynamic changes in web pages (such as changes in element IDs and asynchronous loading of content), and scripts need to be updated frequently to adapt to changes in web page functions.

[0003] Rule-based intelligent operating systems encounter challenges in processing complex user instructions. Such systems usually require pre-definition of detailed task decomposition logic and operation rules, and are difficult to accurately execute tasks that require multiple conditional judgments (such as "booking the cheapest hotel with a rating of 4 stars or above"). At the same time, these systems have limited understanding of web page functions and have a certain degree of uncertainty in cross-page operation path planning.

[0004] Existing large model application solutions attempt to use large models to parse user instructions and generate operation codes, but they still face technical problems in practice. These solutions have limited capabilities in handling dynamic modeling of web page structures and have difficulty effectively handling complex hierarchical operations and page status tracking. Due to excessive reliance on HTML text parsing, these solutions often ignore visual features such as icon positions and button styles, which affects the accuracy of operation element recognition. Summary of the invention

[0005] The present invention provides a web page action execution method, device, electronic device and storage medium to solve the defects in the prior art of limited ability to process dynamic modeling of web page structure, difficulty in effectively processing complex hierarchical operations and page status tracking, and due to excessive reliance on HTML text parsing, these solutions often ignore visual features such as icon position and button style, resulting in the accuracy of operation element recognition being affected.

[0006] The present invention provides a web page action execution method, comprising the following steps: Inputting HTML text data of the webpage of the target website and the page screenshot of the webpage into the visual language model to obtain the page function structured analysis result output by the visual language model; Based on the page function structured analysis result, a function node tree is determined, and a task target vector corresponding to the task text is semantically matched with a function description vector to obtain a candidate operation path; the function description vector is obtained by converting the function description text in the function node tree; Based on the candidate operation paths and the matching degree rankings corresponding to the candidate operation paths, a web page action is executed, and a status verification is performed on the execution result corresponding to the web page action to obtain a status verification result. When the status verification result is a verification failure, the function node tree is backtraced to obtain a first operation path. Based on the first operation path, a web page action is executed. When the execution result corresponding to the web page action is a verification success, the execution result of the web page action is displayed.

[0007] According to a web page action execution method provided by the present invention, the state verification of the execution result corresponding to the web page action to obtain the state verification result includes: The task text, the current page screenshot corresponding to the execution result and the function description text are input into the multimodal large model to obtain the state verification result output by the multimodal large model.

[0008] According to a webpage action execution method provided by the present invention, when the state verification result is verification failure, the function node tree is backtracked to obtain a first operation path, including: When the status verification result is verification failure, determining a node function description set of the function node tree and a tree structure level depth; Starting from the leaf node corresponding to the hierarchical depth of the tree structure, the functional description of the parent node is revised upward layer by layer to obtain the first operation path corresponding to the revised target functional node tree.

[0009] According to a web page action execution method provided by the present invention, starting from the leaf node corresponding to the hierarchical depth of the tree structure, the function description of the parent node is revised upward layer by layer to obtain the first operation path corresponding to the revised target function node tree, including: Perform conflict detection on the function descriptions of each sub-node in the function node tree, and input the conflicting first sub-node function description, the second sub-node function description, the HTML text data, and the page screenshot into the visual language model to obtain the target analysis result output by the visual language model; The function node tree is modified based on the target analysis result, and the specific operation function description in the sub-node function description in the function node tree is converted into a task target description to obtain a first function node tree; Deleting duplicate child nodes and / or invalid child nodes in the first function node tree to obtain a second function node tree; Based on the user task execution frequency, the node function descriptions of the second function node tree are prioritized to obtain a third function node tree; The function descriptions of all child nodes under the same parent node in the third function node tree are input into the multimodal large model for semantic aggregation to obtain a high-level semantic description output by the multimodal large model, and the high-level semantic description is used as the function description of the parent node to obtain the first operation path corresponding to the target function node tree.

[0010] According to a web page action execution method provided by the present invention, the state verification is performed on the execution result corresponding to the web page action to obtain the state verification result, and the method further includes: Obtaining the page status of the web page and the waiting operation time; When the page state is unchanged or the waiting operation time is longer than a first threshold, the function node tree is backtracked to obtain a second operation path.

[0011] According to a web page action execution method provided by the present invention, the step of determining the page status includes: Get the current page and the previous page of the current page; Determine a first hash value of the current page and a second hash value of the previous page; Determine a hash comparison result of the first hash value and the second hash value, and when the hash comparison result is less than a second threshold, determine that the page state is the unchanged state; when the hash comparison result is greater than or equal to the second threshold, determine that the page state is the changed state.

[0012] According to a web page action execution method provided by the present invention, the method further includes: When the status verification result is verification failure, obtaining the executed historical operation path; Adjacent nodes adjacent to each node in the historical operation path are determined from the function node tree, and a third operation path is regenerated based on the adjacent nodes.

[0013] According to a webpage action execution method provided by the present invention, when the state verification result is verification failure, the function node tree is backtracked to obtain a first operation path, including: When the status verification result is verification failure, the function node tree is backtracked based on the node confidence of the function node tree to obtain the first operation path.

[0014] The present invention also provides a web page action execution device, comprising the following modules: An input unit, used to input HTML text data of a web page of a target website and a page screenshot of the web page into the visual language model, and obtain a page function structured analysis result output by the visual language model; A semantic matching unit, for determining a function node tree based on the page function structured analysis result, and semantically matching a task target vector corresponding to the task text with a function description vector to obtain a candidate operation path; the function description vector is obtained by converting the function description text in the function node tree; An execution unit is used to execute a web page action based on the candidate operation path and the matching degree ranking corresponding to the candidate operation path, and perform status verification on the execution result corresponding to the web page action to obtain a status verification result. When the status verification result is a verification failure, the function node tree is backtracked to obtain a first operation path, and the web page action is executed based on the first operation path. When the execution result corresponding to the web page action is a verification success, the execution result of the web page action is displayed.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the web page action execution methods described above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the web page action execution method described in any one of the above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned web page action execution methods.

[0018] The web page action execution method, device, electronic device and storage medium provided by the present invention, on the one hand, determine the page function structured analysis results based on the HTML text data of the web page and the page screenshot of the web page, thereby improving the accuracy of element recognition in the page function structured analysis results; on the other hand, determine the function node tree based on the page function structured analysis results, which can effectively handle complex hierarchical operations, and then semantically match the task target vector corresponding to the task text with the function description vector to determine the operation path, further improving the accuracy and reliability of the page state tracking corresponding to the operation path, and converting the user's natural language instructions into structured operation intentions, which is convenient for subsequent semantic matching and automated task generation, thereby improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 This is one of the flow charts of the web page action execution method provided by the present invention.

[0021] Figure 2 It is a schematic diagram of the process of constructing the function node tree provided by the present invention.

[0022] Figure 3 This is the second flow chart of the web page action execution method provided by the present invention.

[0023] Figure 4 It is a structural schematic diagram of the web page action execution device provided by the present invention.

[0024] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or precedence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type.

[0027] Figure 1 is one of the flow charts of the web page action execution method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0028] Step 110, inputting the HTML text data of the web page of the target website and the page screenshot of the web page into the visual language model to obtain the page function structured analysis result output by the visual language model.

[0029] Specifically, the HTML (HyperText Markup Language) text data of the target website's web page and the page screenshot of the web page can be obtained. For example, the target website can be loaded through a headless browser, and the browser developer tool interface can be used to synchronously obtain the HTML text data of the web page and capture the page screenshot of the web page.

[0030] HTML text data is the source code of a web page, which contains all the elements of the web page (such as titles, paragraphs, buttons, forms, etc.) and their properties. A page screenshot is a visual representation of a web page in its current state, capturing the layout, style, and content of the web page in the form of an image. It can help analyze the visual effects of a web page and verify the visibility and status of elements.

[0031] Page screenshots can be obtained by calling a browser driver (such as WebDriver) to perform operations such as clicks, input, scrolling, etc. and capturing them in real time.

[0032] Then, the HTML text data of the target website's web page and the page screenshots of the web page can be input into the Vision-Language Models (VLMs) to obtain the page function structured analysis results output by the Vision-Language Models.

[0033] Here, the page functional structured analysis result refers to the analysis of the HTML text data and visual representation (page screenshot) of the web page, and the extraction of the operable elements (such as buttons, input boxes, links, etc.) in the page and their functional descriptions.

[0034] Step 120, based on the page function structured analysis result, determine the function node tree, and semantically match the task target vector corresponding to the task text with the function description vector to obtain a candidate operation path; the function description vector is obtained based on the function description text in the function node tree.

[0035] Specifically, after obtaining the page function structured analysis result, the operable elements and their function descriptions in the page function structured analysis result may be organized hierarchically to form a function node tree.

[0036] Then, semantic matching is performed between the task target vector corresponding to the task text and the function description vector to obtain a candidate operation path. The task text may be "book the cheapest hotel with a rating of 4 stars or above" or "query XX weather today", etc. Here, semantic matching between the task target vector and the function description vector may be calculated using methods such as cosine similarity and Pearson correlation coefficient, which is not specifically limited in the embodiment of the present invention.

[0037] The function description vector is obtained by converting the function description text in the function node tree. For example, the function description text can be input into a BERT (Bidirectional Encoder Representations from Transformers) model to obtain the function description text output by the BERT model.

[0038] The task target vector may be obtained by using a large model to parse the task text input by the user, that is, natural language task instructions, and converting them.

[0039] It can be understood that based on the results of the page function structured analysis, the function node tree is determined, and the task target vector corresponding to the task text is semantically matched with the function description vector to obtain the candidate operation path, which can effectively integrate the visual information of the page screenshot of the web page and the structural information of the HTML text data, thereby converting the user's ambiguous text instructions into an executable operation sequence.

[0040] Step 130, based on the candidate operation path and the matching degree ranking corresponding to the candidate operation path, execute the web page action, and perform status verification on the execution result corresponding to the web page action to obtain a status verification result. When the status verification result is verification failure, perform path backtracing on the function node tree to obtain a first operation path, execute the web page action based on the first operation path, and when the execution result corresponding to the web page action is verification success, display the execution result of the web page action.

[0041] Specifically, based on the candidate operation paths and the matching degree rankings corresponding to the candidate operation paths, the web page action is executed, that is, the candidate paths are executed according to the matching degree ranking to simulate the user operation.

[0042] Then, the execution result corresponding to the web page action is verified to obtain the state verification result. Here, the execution result corresponding to the web page action can be verified in combination with the multimodal large model. The state verification includes the page jump completion, such as whether the target page title appears, the visibility of key elements, such as the highlighting of the order submission button, and the asynchronous loading state, such as the completion of data table rendering, etc. The embodiment of the present invention does not specifically limit this.

[0043] When the status verification result is verification failure, the function node tree is traced back to obtain the first operation path, and the web page action is executed based on the first operation path. When the execution result corresponding to the web page action is verification passed, the execution result of the web page action is displayed. For example, when the path execution fails, the specific failure reason is generated, and the failure case is recorded for subsequent system optimization.

[0044] In addition, the function node tree can be stored in the function node database, and the function node tree in the process of performing the web page tree analysis and backtracking correction can also be stored in the function node database.

[0045] The method provided by the embodiment of the present invention inputs the HTML text data of the web page of the target website and the page screenshot of the web page into the visual language large model, obtains the page function structured analysis result output by the visual language large model, and then determines the function node tree based on the page function structured analysis result, and semantically matches the task target vector corresponding to the task text with the function description vector to obtain the candidate operation path; the function description vector is obtained by converting the function description text in the function node tree; finally, based on the candidate operation path and the matching degree ranking corresponding to the candidate operation path, the web page action is executed, and the execution result corresponding to the web page action is verified to obtain the status verification result. When the status verification result is that the verification fails, the function node tree is backtracked to obtain the first operation path, and the web page action is executed based on the first operation path. When the execution result corresponding to the web page action is that the verification passes, the execution result of the web page action is displayed. On the one hand, based on the HTML text data of the web page and the page screenshots of the web page, the page function structured analysis results are determined, which improves the accuracy of element recognition in the page function structured analysis results; on the other hand, based on the page function structured analysis results, the function node tree is determined, which can effectively handle complex hierarchical operations, and then the task target vector corresponding to the task text is semantically matched with the function description vector to determine the operation path, further improving the accuracy and reliability of the page state tracking corresponding to the operation path, and converting the user's natural language instructions into structured operation intentions, which facilitates subsequent semantic matching and automated task generation, thereby improving efficiency.

[0046] Based on the above embodiment, the state verification of the execution result corresponding to the webpage action in step 130 to obtain the state verification result includes: Step 131, input the task text, the current page screenshot corresponding to the execution result and the function description text into the multimodal large model to obtain the state verification result output by the multimodal large model.

[0047] Specifically, the task text, the current page screenshot corresponding to the execution result, and the function description text can be input into the multimodal large model to obtain the state verification result output by the multimodal large model. The multimodal large model can be CLIP (Contrastive Language–Image Pretraining), etc., which is not specifically limited in the embodiment of the present invention.

[0048] It is understandable that the task text provides the background and requirements, the function description text clarifies the goal, and the page screenshot provides intuitive visual information. The multimodal large model can integrate this information to better understand complex scenarios. Furthermore, the multimodal model can match and compare the semantic information in the task text and function description text with the visual information in the current page screenshot, such as verifying whether the button is displayed as required, whether the text is displayed correctly, etc., so as to more comprehensively evaluate the status.

[0049] It should be noted that traditional manual verification methods require a lot of time and effort, while multimodal large models can quickly process text and image information and automatically verify whether functions meet expectations, greatly saving time and labor costs.

[0050] The method provided by the embodiment of the present invention adopts an interactive dynamic execution mechanism to decompose the complex web page automation execution task into multiple operation steps, obtains page screenshots and HTML text data at each key node, verifies the corresponding operation status through a multimodal large model, and obtains a status verification result, thereby significantly improving the intelligence level of the web page automation system, providing an innovative solution for the automatic execution of complex web page tasks, and improving the efficiency of human-computer interaction.

[0051] Based on the above embodiment, in step 130, when the status verification result is verification failure, performing path backtracking on the function node tree to obtain a first operation path includes: Step 210, when the status verification result is verification failure, determining the node function description set of the function node tree and the tree structure level depth; Step 220 , starting from the leaf node corresponding to the hierarchical depth of the tree structure, the functional description of the parent node is revised upward layer by layer to obtain the first operation path corresponding to the revised target functional node tree.

[0052] Specifically, when the status verification result is verification failure, the node function description set {Dleaf} of the function node tree and the tree structure level depth N (preset value, such as N=5) are determined. Here, recursive traversal: the analysis process of the function node tree is repeated for the new page generated by the clickable element until the preset exploration depth is reached.

[0053] Then, starting from the leaf node corresponding to the tree structure level depth N, the functional description of the parent node is revised upward layer by layer to obtain the first operation path corresponding to the revised target functional node tree.

[0054] For example, the subnode function description: Enter your email address; Enter password; Click the "Login" button; Function description of the parent node after aggregation: complete user login authentication.

[0055] Based on the above embodiment, the step 220 takes the leaf node corresponding to the tree structure level depth as the starting point, and modifies the function description of the parent node layer by layer to obtain the first operation path corresponding to the modified target function node tree, including: Step 221, performing conflict detection on the function descriptions of each sub-node in the function node tree, and inputting the conflicting first sub-node function description, the second sub-node function description, the HTML text data, and the page screenshot into the visual language big model to obtain the target analysis result output by the visual language big model; Step 222, based on the target analysis result, the function node tree is modified, and the specific operation function description in the sub-node function description in the function node tree is converted into a task target description to obtain a first function node tree; Step 223, deleting duplicate child nodes and / or invalid child nodes in the first function node tree to obtain a second function node tree; Step 224: based on the user task execution frequency, prioritize the node function descriptions of the second function node tree to obtain a third function node tree; Step 225, input the function descriptions of all child nodes under the same parent node in the third function node tree into the multimodal large model for semantic aggregation, obtain the high-level semantic description output by the multimodal large model, and use the high-level semantic description as the function description of the parent node to obtain the first operation path corresponding to the target function node tree.

[0056] Specifically, a conflict detection is performed on the function description of each sub-node in the function node tree. For example, if there is a contradiction in the function description of the sub-node (such as "submit order" and "cancel order"), the conflicting first sub-node function description, the second sub-node function description, HTML text data, and the page screenshot are input into the visual language model to obtain the target analysis result output by the visual language model. The visual language model will analyze the page screenshot and HTML text data of the web page to determine the actual executable operations. According to the output results (target analysis results) of the visual language model, high-frequency operations or main process-related functions are retained first (such as "submit order" is retained first) to resolve the conflict.

[0057] Then, based on the target analysis results, the function node tree is modified, and the specific operation function description in the sub-node function description in the function node tree is converted into the task target description to obtain the first function node tree. For example, clicking the "search" button is generalized to "perform flight search" to make the function description more universal.

[0058] Furthermore, the duplicate subnodes and / or invalid subnodes in the first function node tree are deleted to obtain a second function node tree. For example, the duplicate subnodes of the first function node tree can be deleted to obtain a second function node tree, the invalid subnodes of the first function node tree can be deleted to obtain a second function node tree, and the duplicate subnodes and invalid subnodes of the first function node tree can also be deleted to obtain a second function node tree. The embodiment of the present invention does not specifically limit this. For example, the repeated pop-up windows generated by page refresh can be deleted. This process helps to reduce redundant information and optimize function descriptions.

[0059] Furthermore, the node function descriptions of the second function node tree can be prioritized based on the user's task execution frequency to obtain a third function node tree. For example, the user's operation path and task execution frequency in the system are recorded. The node function descriptions are prioritized according to the task execution frequency, and a weight is assigned to each node based on the priority ranking. High-frequency task execution operations or key task nodes with higher priority rankings will be given higher weights. In addition, the weight of the node can be dynamically adjusted according to the user's real-time behavior or historical data to ensure that the function node tree can reflect the user's real needs.

[0060] For example, the output format of the third function node tree may be: {"node_id": "P1", "children": ["C1", "C2", "C3"], "function_desc": "Complete the user registration process", "confidence_score": 0.93}; node_id: Unique identifier of the node.

[0061] children: A list of identifiers of child nodes.

[0062] function_desc: Corrected function description.

[0063] confidence_score: The confidence of the model in the feature description.

[0064] Finally, the function descriptions of all child nodes under the same parent node in the third function node tree are input into the multimodal large model for semantic aggregation to obtain a high-level semantic description output by the multimodal large model, and the high-level semantic description is used as the function description of the parent node to obtain the first operation path corresponding to the target function node tree.

[0065] It can be understood that semantic constraints are performed in the operation path planning stage to ensure the accuracy of the operation path, and the method provided by the embodiment of the present invention realizes dynamic correction of functional nodes through bottom-up semantic aggregation and multimodal conflict resolution.

[0066] The method provided by the embodiment of the present invention, 1) through conflict detection, can find the contradictions or inconsistencies between the function descriptions, and use the visual language big model to analyze and correct them, which ensures the accuracy of the function description and avoids user misunderstanding or system errors caused by description conflicts. 2) Convert the specific operation function description into the task goal description, so that the function node tree is closer to the real needs of users, help users quickly understand the functions and operation paths of the system, and improve user satisfaction. 3) Sort the function descriptions according to the frequency of user task execution to ensure that high-frequency tasks are easier to access and execute, thereby improving the response speed and efficiency of the system and reducing user waiting time. 4) By deleting duplicate and invalid child nodes, the redundancy of the function node tree is reduced, making the system structure clearer and facilitating subsequent maintenance and expansion. 5) Through the multimodal big model, the child node function description is semantically aggregated to generate a high-level parent node description, which can more accurately reflect the overall goal and logical structure of the function and avoid the fragmentation of local descriptions affecting the overall understanding.

[0067] Based on the above embodiment, the step 130 of performing status verification on the execution result corresponding to the webpage action to obtain the status verification result also includes: Step 310, obtaining the page status of the web page and the waiting operation time; Step 320: When the page state is unchanged or the waiting operation time is greater than a first threshold, backtrack the function node tree to obtain a second operation path.

[0068] Specifically, the page status of the webpage and the waiting operation duration are obtained, wherein the page status includes an unchanged state and a changed state, and the page status can be determined based on a hash value between the current page and the previous page of the current page.

[0069] When the page state is unchanged or the waiting operation time is longer than the first threshold, the function node tree is backtracked to obtain a second operation path. The first threshold may be 5 seconds or 6 seconds, etc., which is not specifically limited in the embodiment of the present invention.

[0070] Based on the above embodiment, the step of determining the page status includes: Step 410, obtaining the current page and the previous page of the current page; Step 420, determining a first hash value of the current page and a second hash value of the previous page; Step 430, determine a hash comparison result of the first hash value and the second hash value, and when the hash comparison result is less than a second threshold, determine that the page state is the unchanged state; when the hash comparison result is greater than or equal to the second threshold, determine that the page state is the changed state.

[0071] Specifically, the current page and the previous page of the current page are obtained. For example, a browser driver (such as WebDriver) can be called to perform operations such as clicking, inputting, scrolling, etc., and the current page can be captured in real time.

[0072] Then, the first hash value of the current page and the second hash value of the previous page are determined. The hash comparison result of the first hash value and the second hash value can be determined by the Hamming distance, and when the hash comparison result is less than the second threshold, the page state is determined to be unchanged, and when the hash comparison result is greater than or equal to the second threshold, the page state is determined to be changed.

[0073] The method provided by the embodiment of the present invention generally has a lower computational complexity for generating hash values ​​and calculating the Hamming distance. Compared with directly comparing the complete contents of the pages, this method can significantly reduce the consumption of computing resources.

[0074] Based on the above embodiment, the method further includes: Step 510, when the status verification result is verification failure, obtaining the executed historical operation path; Step 520: determine, from the function node tree, adjacent nodes adjacent to each node in the historical operation path, and regenerate a third operation path based on the adjacent nodes.

[0075] Specifically, when the status verification result is that the verification fails, the executed historical operation path is obtained.

[0076] Then, adjacent nodes adjacent to each node in the historical operation path may be determined from the function node tree, and the third operation path may be regenerated based on the adjacent nodes.

[0077] For example, determine the location of the failed node. For example, assume that the current operation path is [P1 ->C1 ->C1.1], and the C1.1 operation fails. Check the parent node C1 of the node, and its sibling nodes (i.e., other child nodes of C1).

[0078] For example, if C1 has child nodes C1.1 and C1.2, and C1.1 fails, try C1.2. If there are no untried nodes at the current level, go back to the previous level and repeat the above process.

[0079] The method provided by the embodiment of the present invention obtains the executed historical operation path when the status verification result is verification failure, and then determines the adjacent nodes adjacent to each node in the historical operation path from the function node tree, and regenerates the third operation path based on the adjacent nodes. By automatically adjusting the operation path, the need for manual intervention by the user is reduced, the user experience is improved, and by regenerating the path, the system can find a more optimal operation path and improve the efficiency of task execution.

[0080] Based on the above embodiment, in step 130, when the status verification result is verification failure, performing path backtracking on the function node tree to obtain a first operation path includes: When the status verification result is verification failure, the function node tree is backtracked based on the node confidence of the function node tree to obtain the first operation path.

[0081] Specifically, when the status verification result is verification failure, the function node tree is backtracked based on the node confidence of the function node tree to obtain the first operation path. For example, function nodes with node confidence (confidence_score) higher than a threshold (such as 0.85) are preferentially selected to generate operation paths. In addition, for low-confidence nodes, the result verification is strengthened after the operation is performed (such as repeating the screenshot analysis 3 times).

[0082] Based on any of the above embodiments, a web page action execution method can be implemented based on a web page action execution system, the system comprising: 1. User interface module: receives user task text (such as "query XX today's weather") and returns the operation result or error reason (such as "weather query function is temporarily unavailable").

[0083] 2. Autonomous exploration module: performs tree analysis and backtracking correction of web pages to generate a functional node database.

[0084] 3. Task processing module: parses task semantics based on the big model, matches functional nodes and generates operation paths.

[0085] 4. Operation execution module: calls the browser driver (such as WebDriver) to perform operations such as click, input, scroll, etc., and captures the page status in real time.

[0086] 5. Verification and backtracking module: Use a multimodal large model to determine whether the operation results meet expectations, and trigger path backtracking when failure occurs.

[0087] 6. Database: stores function node tree, operation history and user task context.

[0088] Based on any of the above embodiments, Figure 2It is a flow chart of constructing the function node tree provided by the present invention, such as Figure 2 As shown, first, the HTML text data of the target website's web page and the page screenshot of the web page are obtained, and then the HTML text data of the target website's web page and the page screenshot of the web page are input into the visual language big model for multimodal semantic analysis, and the page function structured analysis result output by the visual language big model is obtained, and based on the page function structured analysis result, the interactive element function description is generated, so as to construct a function node tree based on the interactive element function description.

[0089] Furthermore, the function node tree is recursively traversed through the child nodes, the semantic description of the parent node is further modified, and then backtracked from the bottom to the top to store the function node tree in the graph database.

[0090] Based on any of the above embodiments, Figure 3 FIG. 2 is a flow chart of the method for executing web page actions provided by the present invention. Figure 3 As shown in the figure, first, natural language instructions, i.e. user task input, are received, and the user task input is converted into a target vector through a large model. Then the function node tree is obtained from the database, and the target vector corresponding to the user task input is semantically matched with the function description vector to obtain a candidate operation path.

[0091] Based on the candidate operation paths and the matching degree ranking (priority) corresponding to the candidate operation paths, the web page action is executed, and based on the new page state, the execution result corresponding to the web page action is multimodal state verification to obtain the state verification result. When the state verification result is that the verification fails, the function node tree is backtracked to obtain the first operation path, that is, exception processing is performed and an alternative path is tried, that is, the failed operation path is replaced with the first operation path, and the web page action is executed based on the first operation path. When the execution result corresponding to the web page action is that the verification passes, the execution result of the web page action is displayed.

[0092] The method provided by the embodiment of the present invention can enhance the intelligence level of human-computer interaction by improving web page automation technology, and provide more effective technical support for the automatic execution of complex web page tasks.

[0093] In summary, (1) a comprehensive web page autonomous exploration mechanism is designed, which can dynamically construct a website function node tree.

[0094] (2) A semantic alignment method based on a multimodal large model is proposed to improve the accuracy of operation element recognition.

[0095] (3) Innovatively apply large models to complex task decomposition and path planning to achieve intelligent web page operations.

[0096] The method provided by the embodiment of the present invention can significantly improve the intelligence level of the web page automation system and provide a new solution for the automatic execution of complex web page tasks.

[0097] The web page action execution device provided by the present invention is described below. The web page action execution device described below and the web page action execution method described above can be referenced to each other.

[0098] Based on any of the above embodiments, the present invention provides a web page action execution device, Figure 4 is a schematic diagram of the structure of the web page action execution device provided by the present invention, such as Figure 4 As shown, the device comprises: Input unit 01, used to input HTML text data of a web page of a target website and a page screenshot of the web page into a visual language model, and obtain a page function structured analysis result output by the visual language model; Semantic matching unit 02, used to determine the function node tree based on the page function structured analysis result, and semantically match the task target vector corresponding to the task text with the function description vector to obtain a candidate operation path; the function description vector is obtained by converting the function description text in the function node tree; The execution unit 03 is used to execute the web page action based on the candidate operation path and the matching degree ranking corresponding to the candidate operation path, and perform status verification on the execution result corresponding to the web page action to obtain the status verification result. When the status verification result is that the verification fails, the function node tree is backtracked to obtain the first operation path, and the web page action is executed based on the first operation path. When the execution result corresponding to the web page action is that the verification passes, the execution result of the web page action is displayed.

[0099] The device provided by the embodiment of the present invention, on the one hand, determines the page function structured analysis results based on the HTML text data of the web page and the page screenshot of the web page, thereby improving the accuracy of element recognition in the page function structured analysis results; on the other hand, based on the page function structured analysis results, determines the function node tree, which can effectively handle complex hierarchical operations, and then semantically matches the task target vector corresponding to the task text with the function description vector to determine the operation path, further improving the accuracy and reliability of the page state tracking corresponding to the operation path, and converts the user's natural language instructions into structured operation intentions, which is convenient for subsequent semantic matching and automated task generation, thereby improving efficiency.

[0100] Based on any of the above embodiments, the execution unit 03 is specifically used for: The task text, the current page screenshot corresponding to the execution result and the function description text are input into the multimodal large model to obtain the state verification result output by the multimodal large model.

[0101] Based on any of the above embodiments, the execution unit 03 specifically includes: A determination unit, configured to determine a node function description set of the function node tree and a tree structure level depth when the status verification result is verification failure; The correction unit is used to correct the function description of the parent node layer by layer, starting from the leaf node corresponding to the hierarchical depth of the tree structure, to obtain the first operation path corresponding to the corrected target function node tree.

[0102] Based on any of the above embodiments, the correction unit is specifically used to: Perform conflict detection on the function descriptions of each sub-node in the function node tree, and input the conflicting first sub-node function description, the second sub-node function description, the HTML text data, and the page screenshot into the visual language model to obtain the target analysis result output by the visual language model; The function node tree is modified based on the target analysis result, and the specific operation function description in the sub-node function description in the function node tree is converted into a task target description to obtain a first function node tree; Deleting duplicate child nodes and / or invalid child nodes in the first function node tree to obtain a second function node tree; Based on the user task execution frequency, the node function descriptions of the second function node tree are prioritized to obtain a third function node tree; The function descriptions of all child nodes under the same parent node in the third function node tree are input into the multimodal large model for semantic aggregation to obtain a high-level semantic description output by the multimodal large model, and the high-level semantic description is used as the function description of the parent node to obtain the first operation path corresponding to the target function node tree.

[0103] Based on any of the above embodiments, a path tracing unit is further included, and the path tracing unit is specifically used to: Obtaining the page status of the web page and the waiting operation time; When the page state is unchanged or the waiting operation time is longer than a first threshold, the function node tree is backtracked to obtain a second operation path.

[0104] Based on any of the above embodiments, it further includes a page status determination unit, and the page status determination unit is specifically used to: Get the current page and the previous page of the current page; Determine a first hash value of the current page and a second hash value of the previous page; Determine a hash comparison result of the first hash value and the second hash value, and when the hash comparison result is less than a second threshold, determine that the page state is the unchanged state; when the hash comparison result is greater than or equal to the second threshold, determine that the page state is the changed state.

[0105] Based on any of the above embodiments, it further includes an operation path generating unit, wherein the operation path generating unit is specifically used to: When the status verification result is verification failure, obtaining the executed historical operation path; Adjacent nodes adjacent to each node in the historical operation path are determined from the function node tree, and a third operation path is regenerated based on the adjacent nodes.

[0106] Based on any of the above embodiments, the execution unit 03 is specifically used for: When the status verification result is verification failure, the function node tree is backtracked based on the node confidence of the function node tree to obtain the first operation path.

[0107] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute a web page action execution method, which includes: inputting the HTML text data of the web page of the target website and the page screenshot of the web page into the visual language large model to obtain the page function structured analysis result output by the visual language large model; based on the page function structured analysis result, determining the function node tree, and semantically matching the task target vector corresponding to the task text with the function description vector to obtain a candidate operation path; the function description vector is obtained by converting the function description text based on the function node tree; based on the candidate operation path and the matching degree ranking corresponding to the candidate operation path, executing the web page action, and performing status verification on the execution result corresponding to the web page action to obtain a status verification result, and when the status verification result is verification failure, performing path backtracking on the function node tree to obtain a first operation path, executing the web page action based on the first operation path, and when the execution result corresponding to the web page action is verification passed, displaying the execution result of the web page action.

[0108] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0109] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the web page action execution method provided by the above methods, the method comprising: inputting the HTML text data of the web page of the target website and the page screenshot of the web page into the visual language large model to obtain the page function structured analysis result output by the visual language large model; based on the page function structured analysis result, determining the function node tree, and semantically matching the task target vector corresponding to the task text with the function description vector to obtain a candidate operation path; the function description vector is obtained by converting the function description text in the function node tree; based on the candidate operation path and the matching degree ranking corresponding to the candidate operation path, executing the web page action, and performing status verification on the execution result corresponding to the web page action to obtain a status verification result, and when the status verification result is verification failure, performing path backtracking on the function node tree to obtain a first operation path, executing the web page action based on the first operation path, and when the execution result corresponding to the web page action is verification passed, displaying the execution result of the web page action.

[0110] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the web page action execution method provided by the above-mentioned methods, the method comprising: inputting the HTML text data of the web page of the target website and the page screenshot of the web page into the visual language large model to obtain the page function structured analysis result output by the visual language large model; based on the page function structured analysis result, determining the function node tree, and semantically matching the task target vector corresponding to the task text with the function description vector to obtain a candidate operation path; the function description vector is obtained by converting the function description text based on the function node tree; based on the candidate operation path and the matching degree ranking corresponding to the candidate operation path, executing the web page action, and performing status verification on the execution result corresponding to the web page action to obtain a status verification result, and when the status verification result is verification failure, performing path backtracking on the function node tree to obtain a first operation path, executing the web page action based on the first operation path, and when the execution result corresponding to the web page action is verification passed, displaying the execution result of the web page action.

[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A web page action execution method, characterized in that: include: Inputting HTML text data of the webpage of the target website and the page screenshot of the webpage into the visual language model to obtain the page function structured analysis result output by the visual language model; Based on the page function structured analysis result, a function node tree is determined, and a task target vector corresponding to the task text is semantically matched with a function description vector to obtain a candidate operation path; the function description vector is obtained by converting the function description text in the function node tree; Based on the candidate operation paths and the matching degree rankings corresponding to the candidate operation paths, a web page action is executed, and a status verification is performed on the execution result corresponding to the web page action to obtain a status verification result. When the status verification result is a verification failure, the function node tree is backtraced to obtain a first operation path. Based on the first operation path, a web page action is executed. When the execution result corresponding to the web page action is a verification success, the execution result of the web page action is displayed.

2. The web page action execution method according to claim 1, characterized in that: The performing status verification on the execution result corresponding to the webpage action to obtain the status verification result includes: The task text, the current page screenshot corresponding to the execution result and the function description text are input into the multimodal large model to obtain the state verification result output by the multimodal large model.

3. The web page action execution method according to claim 1, characterized in that: When the state verification result is verification failure, performing path backtracking on the function node tree to obtain a first operation path includes: When the status verification result is verification failure, determining a node function description set of the function node tree and a tree structure level depth; Starting from the leaf node corresponding to the hierarchical depth of the tree structure, the functional description of the parent node is revised upward layer by layer to obtain the first operation path corresponding to the revised target functional node tree.

4. The web page action execution method according to claim 3, characterized in that: The step of taking the leaf node corresponding to the hierarchical depth of the tree structure as the starting point and revising the function description of the parent node layer by layer to obtain the first operation path corresponding to the revised target function node tree includes: Perform conflict detection on the function descriptions of each sub-node in the function node tree, and input the conflicting first sub-node function description, the second sub-node function description, the HTML text data, and the page screenshot into the visual language model to obtain the target analysis result output by the visual language model; The function node tree is modified based on the target analysis result, and the specific operation function description in the sub-node function description in the function node tree is converted into a task target description to obtain a first function node tree; Deleting duplicate child nodes and / or invalid child nodes in the first function node tree to obtain a second function node tree; Based on the user task execution frequency, the node function descriptions of the second function node tree are prioritized to obtain a third function node tree; The function descriptions of all child nodes under the same parent node in the third function node tree are input into the multimodal large model for semantic aggregation to obtain a high-level semantic description output by the multimodal large model, and the high-level semantic description is used as the function description of the parent node to obtain the first operation path corresponding to the target function node tree.

5. The web page action execution method according to any one of claims 1 to 4, characterized in that: The state verification of the execution result corresponding to the webpage action to obtain the state verification result also includes: Obtaining the page status of the web page and the waiting operation time; When the page state is unchanged or the waiting operation time is longer than a first threshold, the function node tree is backtracked to obtain a second operation path.

6. The web page action execution method according to claim 5, characterized in that: The step of determining the page status includes: Get the current page and the previous page of the current page; Determine a first hash value of the current page and a second hash value of the previous page; Determine a hash comparison result of the first hash value and the second hash value, and when the hash comparison result is less than a second threshold, determine that the page state is the unchanged state; when the hash comparison result is greater than or equal to the second threshold, determine that the page state is the changed state.

7. The web page action execution method according to any one of claims 1 to 4, characterized in that: The method further comprises: When the status verification result is verification failure, obtaining the executed historical operation path; Adjacent nodes adjacent to each node in the historical operation path are determined from the function node tree, and a third operation path is regenerated based on the adjacent nodes.

8. The web page action execution method according to any one of claims 1 to 4, characterized in that: When the state verification result is verification failure, performing path backtracking on the function node tree to obtain a first operation path includes: When the status verification result is verification failure, the function node tree is backtracked based on the node confidence of the function node tree to obtain the first operation path.

9. A web page action execution device, characterized in that: include: An input unit, used to input HTML text data of a web page of a target website and a page screenshot of the web page into the visual language model, and obtain a page function structured analysis result output by the visual language model; A semantic matching unit, for determining a function node tree based on the page function structured analysis result, and semantically matching a task target vector corresponding to the task text with a function description vector to obtain a candidate operation path; the function description vector is obtained by converting the function description text in the function node tree; An execution unit is used to execute a web page action based on the candidate operation path and the matching degree ranking corresponding to the candidate operation path, and perform status verification on the execution result corresponding to the web page action to obtain a status verification result. When the status verification result is a verification failure, the function node tree is backtracked to obtain a first operation path, and the web page action is executed based on the first operation path. When the execution result corresponding to the web page action is a verification success, the execution result of the web page action is displayed.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the web page action execution method according to any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the web page action execution method according to any one of claims 1 to 8 is implemented.

Images