Page interaction method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of computers, and discloses a page interaction method and device, electronic equipment, a storage medium and a program product.The method comprises the steps that a natural language instruction is received to indicate a first model to determine an interaction task chain according to the natural language instruction, and the interaction task chain is sent to a second model; the second model is used for determining an interaction action corresponding to the current interaction task in the interaction task chain; and circularly executing interaction processing until the interaction task chain is ended. By receiving the natural language instruction, the first model automatically generates the interaction task chain, the second model generates the interaction action for the current interaction task according to the interactive page element, and the interaction processing is automatically performed according to the interaction action, so that the problems of lack of universal adaptability and weak webpage content information perception ability in the related technology are solved. Through the two models, respective tasks are completed respectively, and the operation accuracy of actions required by each step in a composite intention or a complex task is enhanced through division of labor and cooperation.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to page interaction methods, devices, electronic devices, storage media, and program products. Background Technology

[0002] In current mainstream browser interaction modes, users need to manually operate through physical input devices. To improve efficiency, script automation solutions are commonly used. This involves writing browser extensions using scripts to precisely analyze the HTML (Hypertext Markup Language) structure of the target webpage and locate the specific page elements that need to be manipulated. Next, developers need to write code to simulate events such as mouse clicks or keyboard input. Finally, the script is deployed and executed to automatically complete a pre-defined, highly specific sequence of tasks. However, automation scripts need to be hard-coded for a single website or a fixed process, and cannot adaptively migrate to new pages with similar structures but different elements based on existing operational experience. They also lack general adaptability to dynamic webpage updates. Summary of the Invention

[0003] This invention provides a page interaction method, device, electronic device, storage medium, and program product to solve the problem that automated scripts cannot adaptively migrate to new pages with similar structures but different elements based on existing operational experience, and lack universal adaptability in the face of dynamic web page updates.

[0004] In a first aspect, the present invention provides a page interaction method, comprising: The system receives natural language instructions to instruct a first model to determine an interaction task chain containing at least one interaction task based on the natural language instructions, and sends the interaction task chain to a second model; the second model is used to determine the interaction action corresponding to the current interaction task in the interaction task chain; the interaction action is used to manipulate target page elements. The interactive processing is executed repeatedly until the interactive task chain ends. The interaction processing includes: Obtain the interactive action corresponding to the current interactive task determined by the second model; Execute the interactive action corresponding to the current interactive task on the target page element in the current page; Extract the first interactive page element from the current page after the interactive action has been performed, and send the first page element set containing each of the first interactive page elements to the second model; the first page element set is used to enable the second model to determine the interactive action corresponding to the next interactive task based on the first page element set.

[0005] The page interaction method provided in this embodiment receives natural language instructions, enabling a first model to automatically generate an interaction task chain based on these instructions. A second model then generates corresponding interactive actions for the current interactive task within the chain, allowing the browser to automatically process the interaction based on these actions. Furthermore, the generation of interactive actions can be based on interactive page elements, addressing the lack of universal adaptability in related technologies when web pages are dynamically updated. Additionally, the extraction of page elements from the webpage overcomes the weakness in perceiving webpage content information caused by visual screenshot analysis in related technologies. By having two models each complete their respective tasks collaboratively, the accuracy of operations for the required actions at each step in complex tasks or with complex intentions is enhanced.

[0006] In some alternative implementations, the method further includes, prior to performing the first round of interaction processing: Extract the second interactive page element from the current page and generate a second page element set containing each of the second interactive page elements.

[0007] The second page element set is sent to the second model. The second page element set is used by the second model to determine the interaction action corresponding to the current interaction task in the first round of interaction processing.

[0008] In some alternative implementations, the first set of page elements is used for: The second model determines, based on the element set of the first page, whether the execution result after the completion of the current interactive task matches the execution expectation corresponding to the current interactive task.

[0009] If the execution result meets the expected result, the second model determines the interaction action corresponding to the next interaction task based on the element set of the first page.

[0010] In some alternative implementations, the first page element set is also used for: If the execution result does not meet expectations, the first model or the second model updates the interaction task chain based on the first page element set.

[0011] The second model determines the interactive action corresponding to the current interactive task in the updated interactive task chain based on the element set of the first page.

[0012] In some alternative implementations, if the execution result does not meet expectations, the method further includes: If the second model cannot determine the interaction action corresponding to the current interaction task in the updated interaction task chain based on the element set of the first page, it waits for the current page to be updated.

[0013] If the current page is updated, extract the third interactive page elements from the updated current page, and send a third page element set containing all the third interactive page elements to the second model. The third page element set is used by the second model to determine the interactive action corresponding to the current interactive task in the updated interactive task chain.

[0014] In some alternative implementations, the first model is also used for: Determine the execution expectations for each interactive task.

[0015] Generate an interactive task chain, which includes various interactive tasks with expected execution, and send the interactive task chain to the second model.

[0016] In some optional implementations, the interaction action includes an interaction method and an interaction object, and performs the interaction action corresponding to the current interaction task on the target page element in the current page, including: Identify the interaction tool corresponding to the interaction method. The interaction tool is used to perform the interaction action.

[0017] Identify the target page element in the current page that corresponds to the interactive object.

[0018] The interactive tools corresponding to the interactive method are used to perform the interactive actions corresponding to the current interactive task on the target page elements.

[0019] In some optional implementations, the first interactive page element in the current page after the interaction action has been performed is extracted, including: Extract the tree structure data of the current page and determine the type tags of each page element in the tree structure data.

[0020] Filter each page element based on its type tag to determine the first interactive page element on the current page.

[0021] In some alternative implementations, sending a first set of page elements, including each of the first interactive page elements, to the second model includes: Determine whether the first interactive page element is obscured by other page elements. If the first interactive page element is not obscured, add the first interactive page element to the first page element set.

[0022] Add a unique element identifier to each of the first interactive page elements in the first page element set.

[0023] The page interaction method provided in this embodiment receives natural language instructions, enabling a first model to automatically generate an interaction task chain based on these instructions. A second model then generates corresponding interactive actions for the current interactive task within the chain. By scheduling interaction tools, steps such as page navigation, element location, data input, and status verification are automatically completed. This reduces manual operations that previously took minutes to just seconds, significantly improving transaction processing efficiency, especially for repetitive and procedural tasks. Furthermore, the interaction actions can be generated based on interactive page elements and their semantic features (such as button text and input box placeholders). This allows the model to understand the universal semantics of buttons like "Search," "Submit," and "Next" on different websites, enabling a single set of instructions to be used across multiple similar scenarios. This demonstrates strong cross-platform and cross-page structure adaptability. Furthermore, by extracting interactive page elements from web pages, redundant HTML or visual information in the original web pages is pre-filtered and refined into a lightweight, semantically clear "set of interactive elements." Compared to existing solutions that directly input the complete DOM tree or page screenshots, this significantly reduces resource consumption and noise interference, and substantially lowers computational resource requirements and inference latency. Secondly, it solves the problem of weak perception of web page content information caused by visual screenshot analysis in related technologies, and enhances the accuracy of operations for the required actions in each step of complex tasks by using the set of interactive page elements as the basis for generating interactive actions.

[0024] In a second aspect, the present invention provides a page interaction device, comprising: The language instruction receiving module receives natural language instructions to instruct the first model to determine an interaction task chain containing at least one interactive task based on the natural language instructions, and then sends the interaction task chain to the second model. The second model determines the interactive action corresponding to the current interactive task in the interaction task chain. The interactive action is used to manipulate target page elements.

[0025] The interaction processing module is used to perform interaction processing in a loop until the interaction task chain ends.

[0026] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the page interaction method of the first aspect or any corresponding embodiment described above.

[0027] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the page interaction method of the first aspect or any corresponding embodiment described above.

[0028] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the page interaction method of the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0029] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 This is a schematic diagram of an application scenario according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the first type of page interaction method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a second type of page interaction method according to an embodiment of the present invention; Figure 4 This is a structural block diagram of a page interaction device according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0033] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0034] As an optional application scenario of this invention, such as Figure 1 As shown, the page interaction system may include at least one terminal device and at least one server. Figure 1 The system is illustrated in the example, which includes a computer 101, a mobile terminal 102, and a server 103, and the terminal devices such as the computer 101 and the mobile terminal 102 are connected to the server 103 through a network 110.

[0035] Specifically, the terminal device can be a smartphone, tablet, laptop, PDA, desktop computer, game console, smart TV, smart wearable device, in-vehicle terminal, VR (Virtual Reality) device, AR (Augmented Reality) device, etc. Server 103 can be a standalone physical server, a server cluster, a distributed system, or a cloud server providing cloud services. Network 110 can be a wired or wireless network, examples of which include, but are not limited to, the Internet, corporate intranet, local area network, wide area network, mobile communication network, and combinations thereof.

[0036] In current mainstream browser interaction modes, users need to manually operate through physical input devices. To improve operational efficiency, script automation solutions are commonly used in related technologies. However, automation scripts need to be hard-coded for a single website or fixed process, and cannot adaptively migrate to new pages with similar structures but different elements based on existing operational experience. They also lack universal adaptability to dynamic webpage updates. There are also technologies that combine AI (Artificial Intelligence) with browser automation. These solutions typically analyze the original HTML structure or visual screenshots of page elements to extract actionable targets, and then use preset rules or model reasoning to determine the next action, thus achieving an initial mapping from language commands to browser operations. However, these technologies also suffer from a lack of semantic understanding and weak perception of webpage content information, making it difficult to accurately define the required actions for each step in cases of complex intents or tasks, resulting in inaccurate operations.

[0037] According to an embodiment of the present invention, a page interaction method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] This embodiment provides a page interaction method that can be used in the aforementioned terminal devices, such as desktop computers, laptops, and mobile phones. Figure 2This is a flowchart of a page interaction method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Receive natural language instructions to instruct the first model to determine an interaction task chain containing at least one interaction task based on the natural language instructions, and send the interaction task chain to the second model. The second model is used to determine the interaction action corresponding to the current interaction task in the interaction task chain. The interaction action is used to manipulate target page elements.

[0039] Natural language instructions can be user-initiated commands input via voice or text. They don't require specialized or procedural vocabulary; instead, they describe the task using everyday language. For example, a natural language instruction could be "Book a flight from Beijing to Shanghai tomorrow." Natural language refers to informal language used by humans in daily communication, such as Chinese and English. It differs from programming languages ​​or machine instructions, characterized by semantic ambiguity, flexible structure, and strong context dependence. After receiving a natural language instruction, it can be sent to the first model. This first model can be an AI model centered around an LLM (Large Language Model), and can also be assisted by other collaborative models, such as visual models, speech models, and retrieval models. It's important to note that the first model is an artificial intelligence (AI) model, not a 3D model specific to the arts. The first model can generate an interactive task chain composed of various interactive tasks based on natural language instructions. Each interactive task chain can include multiple interactive tasks, each task being a step within that chain. These tasks are interconnected, and multiple tasks are linked together to obtain a corresponding interactive task chain. For example, given the instruction "Book a flight from Beijing to Shanghai tomorrow," the generated interactive task chain could be: 1. Redirected to the ticketing website; 2. Enter the departure and destination points, enter the time, and click the search button; 3. Select the corresponding flight from the various flight options displayed on the page and make a reservation.

[0040] Among them, 1, 2 and 3 are all interactive tasks in the interactive task chain.

[0041] After generating the interactive tasks, the interactive task chain can be sent to the second model, which then processes each task sequentially. The second model can also be an artificial intelligence model. By having two models each complete their respective tasks, this division of labor and cooperation improves the accuracy of automated interaction. The second model generates the interactive actions that the browser needs to execute based on the current interactive task in the task chain. The current interactive task refers to the task that needs to be performed in the task chain at the moment. For example, in the interactive task chain mentioned above, if interactive task 1 has been completed, then interactive task 2 is the current interactive task. Each interactive task can generate at least one interactive action. Taking interactive task 2, "Enter the origin and destination, enter the time, and click search," as an example, the interactive actions could include: i. [Enter origin], ii. [Enter destination], iii. [Enter time], iv. [Click button]. The generated interactive actions are used to cause the browser to perform corresponding operations on target page elements in the current page. Page elements are the basic units that constitute webpage content and interactive functions. Page elements can have visual representations (such as buttons, input boxes, and text blocks) or interactive capabilities (such as clickable or inputtable). For example, interactive action iv. [Click the button] means clicking the target page element (i.e., the search button). The method of dividing interactive tasks mentioned above is just one approach; other methods can also be used. Taking "Book a flight from Beijing to Shanghai tomorrow" as an example again, it can be divided into the following six interactive tasks: 1. Redirect to "https: / / flight.xxx.com" and wait for the page to load; 2. Enter "City A" in the departure city input box; 3. Enter "City B" in the destination city input box; 4. Click the date input box and enter the actual date corresponding to "tomorrow"; 5. Click the "Search" button and wait for the results list to render; 6. Select the corresponding flight from the various flight entries displayed on the page and make a reservation.

[0042] Step S202: Execute the interactive processing repeatedly until the interactive task chain ends.

[0043] The interactive processing includes: Step S2021: Obtain the interaction action corresponding to the current interaction task determined by the second model.

[0044] Step S2022: Execute the interactive action corresponding to the current interactive task on the target page element in the current page.

[0045] Step S2023: Extract the first interactive page element from the current page after the interactive action has been performed, and send the first page element set containing all the first interactive page elements to the second model. The first page element set is used by the second model to determine the interactive action corresponding to the next interactive task based on the first page element set.

[0046] After generating the interactive task chain, the interactive processes can be executed cyclically in the browser. For example, the interactive tasks in each task chain can be processed sequentially, meaning there can be a one-to-one correspondence between interactive processes and tasks. Because the interactive task chain has a sequential relationship, the next interactive task's corresponding interactive process continues to be executed after the previous interactive task is completed, until all interactive tasks in the chain have been executed, at which point the interactive task chain ends.

[0047] During the interaction processing of interactive tasks, as mentioned earlier, it is necessary to first obtain the interactive actions corresponding to the current interactive task determined by the second model, and automatically execute the corresponding interactive actions in the browser. The target of the interactive actions is the target page element. After all the interactive actions corresponding to the current interactive task have been executed, the interactive page elements in the current page can be extracted, namely the first interactive page elements. Interactive page elements are those elements on the page that can be interacted with; for example, interactive page elements can be input text boxes, clickable buttons, links, etc. After extracting the first interactive page elements, all first interactive page elements are sent to the second model. For ease of sending, the first interactive page elements can be placed in a set of page elements, namely the first page element set. The first page element set can include all the first interactive page elements. The second model can then determine the interactive action corresponding to the next interactive task based on the first page element set and the next interactive task. If the current interactive task is the last interactive task in the interactive task chain, i.e., there is no next interactive task, the second model may not generate an interactive action.

[0048] The page interaction method provided in this embodiment receives natural language instructions, enabling a first model to automatically generate an interaction task chain based on these instructions. A second model then generates corresponding interactive actions for the current interactive task within the chain, allowing the browser to automatically process the interaction based on these actions. Furthermore, the generation of interactive actions can be based on interactive page elements, addressing the lack of universal adaptability in related technologies when web pages are dynamically updated. Additionally, the extraction of page elements from the webpage overcomes the weakness in perceiving webpage content information caused by visual screenshot analysis in related technologies. By having two models each complete their respective tasks collaboratively, the accuracy of operations for the required actions at each step in complex tasks or with complex intentions is enhanced.

[0049] This embodiment provides a page interaction method that can be used in the aforementioned terminal devices, such as desktop computers, laptops, and mobile phones. Figure 3 This is a flowchart of a page interaction method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps: Step S301: Receive natural language instructions to instruct the first model to determine an interaction task chain containing at least one interaction task based on the natural language instructions, and send the interaction task chain to the second model. The second model is used to determine the interaction action corresponding to the current interaction task in the interaction task chain. The interaction action is used to manipulate target page elements.

[0050] Please see details Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0051] In some alternative implementations, the page interaction method further includes steps S302 and S303 before performing the first round of interaction processing.

[0052] Step S302: Extract the second interactive page elements from the current page and generate a second page element set containing each second interactive page element.

[0053] Step S303: Send the second page element set to the second model. The second page element set is used by the second model to determine the interaction action corresponding to the current interaction task in the first round of interaction processing.

[0054] Understandably, if a page is already open before the first round of interaction processing, i.e., the current page exists, the interactive page elements in the current page can be extracted, i.e., the second interactive page elements, and a corresponding set of second page elements can be generated. After the set of second page elements is sent to the second model, the second model can generate the corresponding interactive action based on the set of second page elements corresponding to the current page, i.e., the interactive action corresponding to the first interactive task in the interaction task chain.

[0055] Step S304: Execute the interactive processing repeatedly until the interactive task chain ends.

[0056] The interactive processing includes: Obtain the interaction action corresponding to the current interaction task determined by the second model.

[0057] Perform the interactive action corresponding to the current interactive task on the target page element in the current page.

[0058] Extract the first interactive page element from the current page after the interaction action has been performed, and send the first page element set containing all the first interactive page elements to the second model. The first page element set is used by the second model to determine the interaction action corresponding to the next interaction task.

[0059] Please see details Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0060] In some optional implementations, the first page element set is used to: enable the second model to determine, based on the first page element set, whether the execution result after the completion of the current interactive task meets the execution expectation corresponding to the current interactive task. If the execution result meets the execution expectation, the second model determines the interactive action corresponding to the next interactive task based on the first page element set.

[0061] After the second model receives the first set of page elements, meaning all interactive actions corresponding to the current interactive task have been completed, it can be determined that the current interactive task has been executed successfully. At this point, the execution result can be determined based on the first set of page elements, and it can be determined whether the execution result meets the execution expectation. The execution expectation is the expected execution result after the interactive task is completed. For example, it could be the minimum page elements that should be present on the page after the current interactive task is completed. Taking the interactive task chain mentioned earlier as an example, after the interactive task "1. Jump to the ticket booking website" is completed, the current page should at least have an input box for entering the destination. That is, the execution expectation can include "an input box for entering the destination". If the execution result of the current interactive task meets the execution expectation corresponding to the current interactive task, it can be determined that the current interactive task has been completed correctly. At this point, the second model can continue to determine the interactive action corresponding to the next interactive task based on the first set of page elements.

[0062] In some optional implementations, the first page element set is further used to: update the interaction task chain based on the first page element set if the execution result does not meet expectations. The second model determines the interaction action corresponding to the current interaction task in the updated interaction task chain based on the first page element set.

[0063] In some cases, the execution result of the current interactive task may not be as expected. Taking the interactive task chain mentioned earlier as an example, after the interactive task "1. Redirect to the ticketing website" is completed, the expected execution may include an "input box for entering the destination". However, there is no corresponding interactive page element in the first page element set. For example, after redirecting to the ticketing website, a "login interface" appears. In this case, it can be determined that the execution result does not meet expectations. At this time, the first model or the second model can update the interactive task chain based on the first page element set. It can be understood that if the first model needs to update the interactive task chain, the first page element set also needs to be sent to the first model.

[0064] For example, after completing the interactive task "1. Redirect to the ticket booking website", the "login interface" appears. At this time, all interactive page elements in the first page element set are also page elements of the "login interface". The first model or the second model can update the interactive task chain based on the first page element set. For example, after the interactive task "1. Redirect to the ticket booking website", an interactive task "2. Log in" can be added. The updated interactive task chain can be: 1. Redirected to the ticketing website; 2. Log in; 3. Enter the departure and destination points, enter the time, and click the search button; 4. Select the corresponding flight from the various flight options displayed on the page and make a reservation.

[0065] If the interactive task "1. Redirect to the ticket booking website" has been completed, then the current interactive task is "2. Log in". The second model can then generate the corresponding interactive action (i.e. "2. Log in") based on the current interactive task and the first page element set. For example, it could be i. [Enter username], ii. [Enter password], or iii. [Click the login button].

[0066] In some optional implementations, if the execution result does not meet expectations, the page interaction method may further include steps a1 and a2.

[0067] Step a1: If the second model cannot determine the interaction action corresponding to the current interaction task in the updated interaction task chain based on the first page element set, wait for the current page to be updated.

[0068] In some cases, for the current interactive task in the updated interactive task chain, the second model may not be able to determine the corresponding interactive action. For example, if the updated current interactive task is "2. Log in," but the second model generates incomplete interactive actions, such as i. [Enter username], ii. [Enter password], and iii. [Click login button], the second model cannot determine the specific username or password that needs to be entered. In this case, it can be determined that the second model cannot determine the corresponding interactive action for the current interactive task in the updated interactive task chain based on the first page element set. In this situation, it is necessary to wait for the current page to update. For example, if the user can manually enter their username and password, then the current page has been updated. Understandably, while waiting for the current page to update, a prompt can be displayed to the user explaining the current situation, such as "User needs to enter username and password."

[0069] Step a2: If the current page is updated, extract the third interactive page elements from the updated current page and send the third page element set containing all the third interactive page elements to the second model. The third page element set is used by the second model to determine the interactive action corresponding to the current interactive task in the updated interactive task chain.

[0070] After the page is updated, the interactive page elements (i.e., the third interactive page elements) in the updated current page can be extracted, and a corresponding third page element set can be generated based on the third interactive page elements and sent to the second model. At this time, the second model can generate the corresponding interactive action for the current interactive task based on the third page element set. For example, the interactive actions corresponding to the current interactive task "2. Log in" are i. [Enter username], ii. [Enter password], and iii. [Click the login button]. At this time, the username or password cannot be entered automatically. After the user manually enters the username and password, the username input page element and password input page element in the third page element set will be filled. Therefore, the interactive action corresponding to the current interactive task "2. Log in" can be i. [Click the login button].

[0071] In some optional implementations, the first model is further configured to: determine the execution expectations corresponding to each interactive task; generate an interactive task chain, which includes each interactive task with an execution expectation; and send the interactive task chain to the second model.

[0072] As mentioned earlier, after the current interactive task is completed, it is necessary to determine whether the execution result of the current interactive task meets the execution expectation of the current interactive task. During the generation of the interactive task chain, the first model can determine the execution expectation of each interactive task while generating each interactive task, and send the interactive task chain with the execution expectation of each interactive task to the second model, so that the second model can determine whether the execution result meets the execution expectation of the current interactive task after the current interactive task is completed.

[0073] In some optional implementations, the interactive action includes the interactive method and the interactive object. Step S304, "execute the interactive action corresponding to the current interactive task on the target page element in the current page", includes steps b1 to b3.

[0074] Step b1: Determine the interaction tool corresponding to the interaction method; the interaction tool is used to perform the interaction action; Step b2: Determine the target page element in the current page that corresponds to the interactive object; Step b3: Use the interaction tool corresponding to the interaction method to perform the interaction action corresponding to the current interaction task on the target page element.

[0075] To make operations more precise, interactive actions can include interaction methods and interaction objects. Interaction methods can be, for example, "jump," "click," or "input," while the interaction object is the object corresponding to the interactive action; it is a specific element on the page, such as a target page element. Furthermore, interactive actions can also include text content. For example, when the interaction method is "input," the interaction object can include a "departure location text box" and the text content "City A," meaning the interactive action could be "Input 'City A' into the 'departure location text box'." After generating the interactive action, the corresponding interactive tool can be determined based on the interaction method within the action to achieve control of the browser window and automated operations.

[0076] During implementation, some interactive tools can be pre-set. Interactive tools are those that the browser can automatically invoke and execute on interactive page elements; that is, interactive tools can translate "intent" into "action." Each interactive tool corresponds to a specific, low-level operation. Interactive tools can include tab-based interactive tools, element-based interactive tools, and function-based interactive tools. Tab-based interactive tools can include tools such as create tab (tab_create), read all tabs (tab_get_all), read currently active tabs (tab_get_active), switch between active tabs (tab_switch), close tabs (tab_close), and get tab content (tab_get_html_content). Element-based interactive tools can include click tools (element_click) and input tools (element_input_text). Function-based interactive tools can include waiting tools (wait).

[0077] For example, the "click" interaction method can use the click tool (element_click) to simulate a mouse click effect. Then, by identifying the interaction object, the target page element corresponding to the interaction object in the current page can be determined. For example, if the interaction object is a "login button," the target page element corresponding to the "login button" can be found in either the first or second page element set, and the corresponding interaction action can be performed on that target page element using the interaction tool corresponding to the interaction method.

[0078] In some optional implementations, step S304, "extracting the first interactive page element in the current page after the interactive action has been performed," includes steps c1 and c2.

[0079] Step c1: Extract the tree structure data of the current page and determine the type tags of each page element in the tree structure data.

[0080] Step c2: Filter each page element according to its type tag to determine the first interactive page element in the current page.

[0081] In the process of extracting the first interactive page, the DOM (Document Object Model) tree (i.e., tree-structured data) of the current page can be extracted first, and the type tags of each element node (i.e., page element) contained in the extracted DOM tree can be determined. Type tags can include, for example, DIV, INPUT, BUTTON, etc. Then, page elements are filtered according to their type tags to identify all page elements with user interactivity. For example, page elements with interactive tag types such as INPUT, BUTTON, and A can be extracted and identified as interactive. Page elements with type tags such as DIV, SPAN, and IMG are usually non-interactive. Excluding non-interactive page elements helps the second model quickly identify the current page and generate corresponding interactive actions.

[0082] Similarly, this method can also be used to extract the second and third interactive page elements, which will not be elaborated upon in this embodiment.

[0083] In some alternative implementations, step S304, "sending a first set of page elements containing each first interactive page element to a second model," includes steps d1 and d2.

[0084] Step d1: Determine whether the first interactive page element is obscured by other page elements. If the first interactive page element is not obscured, add the first interactive page element to the first page element set.

[0085] Step d2: Add a unique element identifier to each of the first interactive page elements in the first page element set.

[0086] In some cases, some first-interactive page elements may be obscured by other page elements. For example, a destination input box might be below an advertisement that needs to be closed. In such cases, it's impossible to interact with the obscured first-interactive page elements. Therefore, during the generation of the first page element set, the unobscured first-interactive page elements can be added to the first page element set first. When determining whether a first-interactive page element is obscured, methods such as `getBoundingClientRect` and `elementFromPoint` can be used to ensure that the first-interactive page element is visible and not obscured by other page elements. After generating the first page element set, a unique element identifier (e.g., element ID) can be added to each first-interactive page element in the first page element set. When the second model generates interactive actions based on the first page element set, the element identifier is used to distinguish different page elements and identify the interactive object. Similarly, this method can be used to generate the second and third page element sets, which will not be elaborated upon in this embodiment.

[0087] The page interaction method provided in this embodiment receives natural language instructions, enabling a first model to automatically generate an interaction task chain based on these instructions. A second model then generates corresponding interactive actions for the current interactive task within the chain. By scheduling interaction tools, steps such as page navigation, element location, data input, and status verification are automatically completed. This reduces manual operations that previously took minutes to just seconds, significantly improving transaction processing efficiency, especially for repetitive and procedural tasks. Furthermore, the interaction actions can be generated based on interactive page elements and their semantic features (such as button text and input box placeholders). This allows the model to understand the universal semantics of buttons like "Search," "Submit," and "Next" on different websites, enabling a single set of instructions to be used across multiple similar scenarios. This demonstrates strong cross-platform and cross-page structure adaptability. Furthermore, by extracting interactive page elements from web pages, redundant HTML or visual information in the original web pages is pre-filtered and refined into a lightweight, semantically clear "set of interactive elements." Compared to existing solutions that directly input the complete DOM tree or page screenshots, this significantly reduces resource consumption and noise interference, and substantially lowers computational resource requirements and inference latency. Secondly, it solves the problem of weak perception of web page content information caused by visual screenshot analysis in related technologies, and enhances the accuracy of operations for the required actions in each step of complex tasks by using the set of interactive page elements as the basis for generating interactive actions.

[0088] This embodiment also provides a page interaction device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0089] This embodiment provides a page interaction device, such as... Figure 4 As shown, it includes: The language instruction receiving module 401 is used to receive natural language instructions, instructing the first model to determine an interaction task chain containing at least one interaction task based on the natural language instructions, and to send the interaction task chain to the second model. The second model is used to determine the interaction action corresponding to the current interaction task in the interaction task chain. The interaction action is used to manipulate target page elements.

[0090] The interaction processing module 402 is used to perform interaction processing in a loop until the interaction task chain ends.

[0091] The interaction processing module 402 includes: The "Get Interaction" submodule is used to obtain the interaction action corresponding to the current interaction task determined by the second model.

[0092] The interactive action execution submodule is used to execute the interactive action corresponding to the current interactive task on the target page element in the current page.

[0093] The interaction action generation submodule extracts the first interactive page element from the current page after the interaction action has been executed, and sends a first page element set containing all the first interactive page elements to the second model. The first page element set is used by the second model to determine the interaction action corresponding to the next interaction task.

[0094] In some alternative implementations, the page interaction device further includes: The page element extraction module is used to extract the second interactive page elements in the current page and generate a second page element set containing each second interactive page element.

[0095] The page element sending module is used to send the second set of page elements to the second model. The second set of page elements is used by the second model to determine the interaction action corresponding to the current interaction task in the first round of interaction processing.

[0096] In some optional implementations, the first page element set is used to: enable the second model to determine, based on the first page element set, whether the execution result after the completion of the current interactive task meets the execution expectation corresponding to the current interactive task. If the execution result meets the execution expectation, the second model determines the interactive action corresponding to the next interactive task based on the first page element set.

[0097] In some optional implementations, the first page element set is further used to: update the interaction task chain based on the first page element set if the execution result does not meet expectations. The second model determines the interaction action corresponding to the current interaction task in the updated interaction task chain based on the first page element set.

[0098] In some optional implementations, if the execution result does not meet expectations, the page interaction device further includes: The "Wait for Page Update" module is used to wait for the current page to be updated when the second model cannot determine the interaction action corresponding to the current interaction task in the updated interaction task chain based on the first page element set.

[0099] The element extraction module is used to extract third-party interactive page elements from the updated current page when the current page is updated, and then send a set of third-party interactive page elements, containing all of these elements, to the second model. This set of third-party interactive page elements enables the second model to determine the interactive action corresponding to the current interactive task in the updated interactive task chain.

[0100] In some optional implementations, the first model is further configured to: determine the execution expectations corresponding to each interactive task; generate an interactive task chain, which includes each interactive task with an execution expectation; and send the interactive task chain to the second model.

[0101] In some optional implementations, the interaction action includes the interaction method and the interaction object, and the interaction action execution submodule includes: The interaction tool determination unit is used to determine the interaction tool corresponding to the interaction method. The interaction tool is used to perform the interaction action.

[0102] The target element determination unit is used to determine the target page element in the current page that corresponds to the interactive object.

[0103] The interactive action execution unit is used to execute the interactive action corresponding to the current interactive task on the target page element through the interactive tool corresponding to the interactive method.

[0104] In some optional implementations, the interactive action generation submodule includes: The tree structure extraction unit is used to extract the tree structure data of the current page and determine the type tags of each page element in the tree structure data.

[0105] The page element filtering unit is used to filter page elements based on their type tags to determine the first interactive page element in the current page.

[0106] In some optional implementations, the interactive action generation submodule includes: The page element occlusion unit is used to determine whether the first interactive page element is occluded by other page elements. If the first interactive page element is not occluded, the first interactive page element is added to the first page element set.

[0107] The element identifier addition unit is used to add a unique element identifier to each first interactive page element in the first page element set.

[0108] The page interaction device provided in this embodiment of the invention can execute the page interaction method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution. Further functional descriptions of the various modules and units described above are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0109] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0110] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0111] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0112] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the page interaction method of the embodiments of the present invention.

[0113] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0114] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the page interaction method shown in the above embodiments is implemented.

[0115] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0116] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A page interaction method, characterized in that, The method includes: The system receives natural language instructions to instruct a first model to determine an interaction task chain containing at least one interaction task based on the natural language instructions, and sends the interaction task chain to a second model; the second model is used to determine the interaction action corresponding to the current interaction task in the interaction task chain; the interaction action is used to manipulate target page elements. The interactive processing is executed repeatedly until the interactive task chain ends. The interaction processing includes: Obtain the interactive action corresponding to the current interactive task determined by the second model; Execute the interactive action corresponding to the current interactive task on the target page element in the current page; Extract the first interactive page element from the current page after the interactive action has been performed, and send the first page element set containing each of the first interactive page elements to the second model; the first page element set is used to enable the second model to determine the interactive action corresponding to the next interactive task based on the first page element set.

2. The method according to claim 1, characterized in that, Before performing the first round of interaction processing, the method further includes: Extract the second interactive page element from the current page and generate a second page element set containing each of the second interactive page elements; The second page element set is sent to the second model; the second page element set is used to enable the second model to determine the interaction action corresponding to the current interaction task in the first round of interaction processing based on the second page element set.

3. The method according to claim 1, characterized in that, The first set of page elements is used for: The second model determines, based on the first set of page elements, whether the execution result after the completion of the current interactive task matches the execution expectation corresponding to the current interactive task. If the execution result meets the expected execution, the second model determines the interaction action corresponding to the next interaction task based on the first set of page elements.

4. The method according to claim 3, characterized in that, The first page element set is also used for: If the execution result does not meet the execution expectation, the first model or the second model shall update the interactive task chain according to the first page element set; The second model determines the interactive action corresponding to the current interactive task in the updated interactive task chain based on the first page element set.

5. The method according to claim 4, characterized in that, If the execution result does not meet the expected execution result, the method further includes: If the second model cannot determine the interaction action corresponding to the current interaction task in the updated interaction task chain based on the first page element set, it waits for the current page to be updated. When the current page is updated, the third interactive page element in the updated current page is extracted, and a third page element set containing each of the third interactive page elements is sent to the second model; the third page element set is used to enable the second model to determine the interactive action corresponding to the current interactive task in the updated interactive task chain based on the third page element set.

6. The method according to claim 3, characterized in that, The first model is also used for: Determine the execution expectation corresponding to each of the aforementioned interactive tasks; Generate the interaction task chain, which includes each interaction task having the execution expectation, and send the interaction task chain to the second model.

7. A page interaction device, characterized in that, The device includes: A language instruction receiving module is used to receive natural language instructions to instruct a first model to determine an interaction task chain containing at least one interaction task based on the natural language instructions, and to send the interaction task chain to a second model; the second model is used to determine the interaction action corresponding to the current interaction task in the interaction task chain; the interaction action is used to operate a target page element; The interaction processing module is used to perform interaction processing in a loop until the interaction task chain ends.

8. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the page interaction method of any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the page interaction method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the page interaction method according to any one of claims 1 to 6.

Images