Webpage action execution method, device, equipment and program product

Abstract

The invention provides a webpage action execution method and device, equipment and a program product, and the webpage action execution method comprises the steps: determining webpage operation information of a next action based on a to-be-executed task, webpage operation information of a current action, a screenshot of the current action, webpage operation information of a previous action and a screenshot of the previous action; and executing the next action based on the webpage operation information of the next action. According to the method, the webpage action execution accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a web page action execution method, device, equipment and program product. Background Art

[0002] With the rapid development of society, the use of the Internet has increased year by year, and various applications have emerged one after another. Intelligent web navigation has become an important challenge facing human-computer interaction technology. Intelligent web navigation needs to automatically execute web page process operations based on user needs. However, users often need to quickly find the required information or complete specific tasks in complex and changing web page structures, such as online shopping, booking services, or obtaining public information, which places high demands on process correctness and reliability.

[0003] Traditional web navigation methods mainly rely on manual user operations and rule-based scripts. These methods are relatively effective in simple scenarios, but they are inefficient, error-prone, and inflexible when dealing with complex or dynamic web pages. They are also extremely sensitive to changes in web design. The real-time, diversity, and complexity of web pages result in low process accuracy and reliability in current web navigation methods. Currently, there is a problem of low accuracy in the execution of web page actions. Summary of the invention

[0004] The present application provides a web page action execution method, device, equipment and program product to solve the problem of low accuracy in web page action execution.

[0005] According to a first aspect of an embodiment of the present application, a method for executing a webpage action is provided, comprising:

[0006] Determine web page operation information for a next action based on the task to be performed, web page operation information for a current action, a screenshot of the current action, web page operation information for a previous action, and a screenshot of the previous action;

[0007] Based on the webpage operation information of the next action, the next action is executed.

[0008] Optionally, determining the webpage operation information of the next action based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, and the screenshot of the previous action includes:

[0009] Determine a next action description based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, and the screenshot of the previous action;

[0010] Based on the next action description, determine candidate web page information for the next action;

[0011] Based on the task to be executed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action, the screenshot of the previous action, the next action description and the candidate web page information of the next action, the web page operation information of the next action is determined.

[0012] Optionally, the determining the next action description based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, and the screenshot of the previous action includes:

[0013] Generate a first prompt word based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, and the screenshot of the previous action;

[0014] Input the first prompt word into the action description generation model to obtain the next action description;

[0015] Among them, the first prompt word is used to instruct the action description generation model to generate the next action description based on the task to be performed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action and the screenshot of the previous action.

[0016] Optionally, determining candidate web page information for the next action based on the next action description includes:

[0017] Determine the key information of each element in the current web page;

[0018] Generate a first vector corresponding to each key information of each element;

[0019] Generate a second vector describing the next step action;

[0020] Based on each of the first vectors and the second vectors, filtering out key information of a target element from key information of each of the elements;

[0021] The key information of the target element is determined as candidate web page information for the next action.

[0022] Optionally, determining the webpage operation information of the next action based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, the screenshot of the previous action, the next action description, and the candidate webpage information of the next action includes:

[0023] Generate a second prompt word based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, the screenshot of the previous action, the next action description, and the candidate webpage information of the next action;

[0024] Inputting the second prompt word into the operation information generation model to obtain the webpage operation information of the next step action;

[0025] Among them, the second prompt word is used to instruct the operation information generation model to generate the web page operation information of the next action based on the task to be executed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action, the screenshot of the previous action, the next action description and the candidate web page information of the next action.

[0026] Optionally, the webpage operation information includes an action and webpage information corresponding to the action;

[0027] The action includes at least one of mouse clicking, mouse hovering, element selection, content typing, web page scrolling up and down, web page forward and backward, web page jumping, and generating task conclusion.

[0028] Optionally, executing the next action based on the webpage operation information of the next action includes:

[0029] Obtaining the action function corresponding to the next step action;

[0030] Based on the web page information corresponding to the next action, the action function corresponding to the next action is executed to implement the next action.

[0031] Optionally, after determining the webpage operation information of the next action based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, the screenshot of the previous action, the next action description, and the candidate webpage information of the next action, the method further includes:

[0032] Generate an evaluation prompt word based on the task to be performed, the webpage operation information of the current action, the webpage operation information of the previous action, the next action description, the candidate webpage information of the next action, and the webpage operation information of the next action;

[0033] Input the evaluation prompt word into the evaluation big model to obtain the evaluation result corresponding to the webpage operation information of the next action;

[0034] Among them, the evaluation prompt word is used to indicate that the evaluation model generates an evaluation result corresponding to the web page operation information of the next action based on the task to be executed, the web page operation information of the current action, the web page operation information of the previous action, the next action description, the candidate web page information of the next action and the web page operation information of the next action.

[0035] Optionally, after inputting the evaluation prompt word into the evaluation macro model to obtain the evaluation result corresponding to the webpage operation information of the next action, the method further includes:

[0036] In the case where the evaluation result corresponding to the webpage operation information of the next action indicates that the webpage operation information of the next action is wrong, generating an error correction prompt word based on the task to be executed, the webpage operation information of the current action, the webpage operation information of the previous action, the next action description, the candidate webpage information of the next action, the webpage operation information of the next action, and the evaluation result corresponding to the webpage operation information of the next action;

[0037] Input the error correction prompt word into the error correction large model to obtain the corrected web page operation information for the next action;

[0038] Among them, the error correction prompt word is used to instruct the error correction model to generate corrected web page operation information for the next action based on the task to be executed, the web page operation information of the current action, the web page operation information of the previous action, the next action description, the candidate web page information of the next action, the web page operation information of the next action, and the evaluation result corresponding to the web page operation information of the next action.

[0039] Optionally, after inputting the error correction prompt word into the error correction large model to obtain the corrected webpage operation information for the next action, the method further includes:

[0040] The web page operation information of the next action is replaced with the corrected web page operation information of the next action, and the step of generating evaluation prompt words based on the task to be executed, the web page operation information of the current action, the web page operation information of the previous action, the next action description, the candidate web page information of the next action and the web page operation information of the next action is returned to, until the evaluation result indicates that the web page operation information is correct.

[0041] According to a second aspect of an embodiment of the present application, a web page action execution device is provided, including:

[0042] A processing unit, configured to determine web page operation information for a next action based on the task to be performed, web page operation information for a current action, a screenshot of the current action, web page operation information for a previous action, and a screenshot of the previous action;

[0043] The execution unit is used to execute the next action based on the webpage operation information of the next action.

[0044] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including a memory and a processor;

[0045] The memory is connected to the processor and is used to store programs;

[0046] The processor is used to implement the web page action execution method as described in the first aspect by running the program in the memory.

[0047] According to a fourth aspect of an embodiment of the present application, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to execute the web page action execution method as described in the first aspect.

[0048] In the present application, based on the task to be executed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action and the screenshot of the previous action, the web page operation information of the next action is determined, the user needs are perceived from the task to be executed, and based on the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action and the screenshot of the previous action, the association between the text information and the image semantic information within the current action, the association between the text information and the image semantic information within the previous action, and the association between the current action and the previous action are fully understood, the changes in the web page are fully learned, and then the web page operation information of the next action is determined, which conforms to the logic of human operation, is accurate and reliable, and improves the accuracy of the web page operation information of the next action. Based on the web page operation information of the next action, the next action is executed. On the basis of improving the accuracy of the web page operation information of the next action, the accuracy of the next action to be executed is further improved, thereby improving the accuracy of the execution of the web page action. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0050] Figure 1A schematic diagram of a flow chart of a web page action execution method provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of a process of step 101 provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of a process of step 201 provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of a process of step 202 provided in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of a process of step 203 provided in an embodiment of the present application;

[0055] Figure 6 A schematic diagram of a flow chart of a web page action execution method provided in an embodiment of the present application;

[0056] Figure 7 A schematic diagram of a flow chart of a web page action execution method provided in an embodiment of the present application;

[0057] Figure 8 A schematic diagram of the structure of a web page action execution device provided in an embodiment of the present application;

[0058] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] With the development of deep learning, many capabilities in the field of human-computer interaction have been significantly improved, such as knowledge question answering, text classification, information extraction, machine translation, etc. These advances provide users with a richer and more convenient interactive experience. Almost all popular smart assistant devices currently use deep learning technology to process and understand users' natural language instructions to achieve more accurate services. And they can adapt to users' language habits and preferences through continuous learning and provide more personalized interactive responses.

[0060] With the rapid development of society, the use of the Internet has increased year by year, and various applications have emerged one after another. Intelligent web navigation has become an important challenge facing human-computer interaction technology. Intelligent web navigation needs to automatically execute web page process operations based on user needs. However, users often need to quickly find the required information or complete specific tasks in complex and changing web page structures, such as online shopping, booking services, or obtaining public information, which places high demands on process correctness and reliability.

[0061] Traditional web navigation methods mainly rely on manual user operations and rule-based scripts. These methods are relatively effective in simple scenarios, but when dealing with complex or dynamic web pages, they are inefficient, have high error rates, lack flexibility, and are extremely sensitive to changes in web design. Existing web navigation algorithms based on deep learning have brought revolutionary improvements, which can understand users' natural language instructions and identify web page elements, thereby achieving smarter user interaction. However, it is still unable to better solve the problems of low process accuracy and reliability caused by the real-time, diversity, and complexity of web pages, and it is also difficult for current algorithms to better understand users' intentions and context. These challenging problems show that although the existing technology has broad prospects, it still faces many difficulties before it can be widely used. There is currently a problem of low accuracy in the execution of web page actions.

[0062] In order to solve the problem of low accuracy in executing web page actions, the present application provides a web page action execution method, apparatus, device and program product.

[0063] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0064] Exemplary Implementation Environment

[0065] The web page action execution method according to the embodiment of the present disclosure can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a cloud server capable of cloud computing. The method can be implemented by a processor calling a computer-readable program instruction stored in a memory.

[0066] Exemplary Methods

[0067] See also Figure 1 In an exemplary embodiment, a method for executing a web page action is provided. Figure 1 As shown, the process of the web page action execution method mainly includes:

[0068] Step 101, based on the task to be executed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action and the screenshot of the previous action, determine the webpage operation information of the next action.

[0069] In an exemplary embodiment, the task to be performed can be a natural language task of the user, for example, please help me search for watches priced less than 50 on application A, help me search for today's hot news on application B, please find song D by author A on music application C, etc.

[0070] In an exemplary embodiment, the webpage operation information includes an action and webpage information corresponding to the action. The webpage operation information can determine the specific operation of the webpage navigation and contains rich text information.

[0071] In an exemplary embodiment, the webpage operation information may be expressed in HTML (HyperText Markup Language); the webpage operation information may also be expressed in natural language.

[0072] In an exemplary embodiment, the current action refers to the action being performed on the web page, the previous action refers to the action performed on the web page before the current action, and the next action refers to the action that should be performed on the web page after the current action. For example, the task to be performed is "help me search for brand E mobile phones with 256G memory and then foldable screen on application A", the web page operation information of the previous action is "mouse click search button", the web page operation information of the current action is "type brand E mobile phone in the search box", and the next action description is "find 256G body memory information".

[0073] In an exemplary embodiment, the screenshot of the current action is a web page interface obtained by executing the current action based on the web page operation information of the current action, and the screenshot of the previous action is a web page interface obtained by executing the previous action based on the web page operation information of the previous action. The screenshots of the current action and the screenshots of the previous action contain rich image semantic information.

[0074] Based on the tasks to be performed, the web page operation information of the current action, the screenshots of the current action, the web page operation information of the previous action and the screenshots of the previous action, the web page operation information of the next action is determined, the user needs are perceived from the tasks to be performed, and based on the web page operation information of the current action, the screenshots of the current action, the web page operation information of the previous action and the screenshots of the previous action, the association between the text information and the image semantic information in the current action, the association between the text information and the image semantic information in the previous action, and the association between the current action and the previous action is fully understood, the changes in the web page are fully learned, and then the web page operation information of the next action is determined, which is in line with the logic of human operations, accurate and reliable, and improves the accuracy of the web page operation information of the next action.

[0075] In some embodiments, step 101 includes:

[0076] Generate a third prompt word based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, and the screenshot of the previous action;

[0077] Input the third prompt word into the large model to obtain web page operation information for the next action;

[0078] Among them, the third prompt word is used to instruct the large model to generate web page operation information for the next action based on the task to be executed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action and the screenshot of the previous action.

[0079] In the exemplary embodiment, a large model (also called a foundation model) refers to a machine learning model with large-scale parameters and complex computing structure, which can process massive data and complete various complex tasks, such as natural language processing, computer vision, speech recognition, etc. Large models usually have billions or even hundreds of billions of parameters.

[0080] In an exemplary embodiment, the big model can be a multimodal big model. For example, the big model is an open source multimodal big model LLaVA-NeXT, which supports multi-image input and demonstrates powerful capabilities in image reasoning, video analysis, 3D scene understanding, and complex visual command tasks.

[0081] In other embodiments, Figure 2 As shown, step 101 includes:

[0082] Step 201, based on the task to be executed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action and the screenshot of the previous action, determine the next action description.

[0083] In an exemplary embodiment, the next action description refers to a natural language description of the next action. For example, the next action description is "search for 256G of body memory information".

[0084] In some embodiments, step 201 includes: extracting image semantic information of the current action from a screenshot of the current action; extracting image semantic information of the previous action from a screenshot of the previous action; determining the next action description based on the task to be performed, the web page operation information of the current action, the image semantic information of the current action, the web page operation information of the previous action, and the image semantic information of the previous action.

[0085] In an exemplary embodiment, the image semantic information of the current action may be the information in the screenshot of the current action expressed in natural language, and the image semantic information of the previous action may be the image semantic information of the previous action expressed in natural language. Based on the task to be performed, the webpage operation information of the current action, the image semantic information of the current action, the webpage operation information of the previous action, and the image semantic information of the previous action, the next action description is determined, and the next action description can be obtained through a large language model based on the task to be performed, the webpage operation information of the current action, the image semantic information of the current action, the webpage operation information of the previous action, and the image semantic information of the previous action.

[0086] In an exemplary embodiment, Large Language Models (LLMs) refer to deep learning models trained using large amounts of text data, and are intended to understand and generate natural language text.

[0087] In other embodiments, Figure 3 As shown, step 201 includes:

[0088] Step 301 : Generate a first prompt word based on the task to be executed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, and the screenshot of the previous action.

[0089] Step 302, input the first prompt word into the action description generation model to obtain the next action description.

[0090] Among them, the first prompt word is used to instruct the action description generation model to generate the next action description based on the task to be executed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action and the screenshot of the previous action.

[0091] In an exemplary embodiment, the action description generation big model can be a multimodal big model. For example, the big model is the open source multimodal big model LLaVA-NeXT, which supports multi-image input and demonstrates powerful capabilities in image reasoning, video analysis, 3D scene understanding and complex visual instruction tasks.

[0092] By generating a large model through action description, we can perceive user needs from the tasks to be performed, fully understand the relationship between the text information and image semantic information in the current action, the relationship between the text information and image semantic information in the previous action, and the relationship between the current action and the previous action, fully learn the changes in the web page, and then infer the next action description, which is in line with the logic of human operation, accurate and reliable, and improves the accuracy of the next action description.

[0093] Step 202: Determine candidate web page information for the next action based on the next action description.

[0094] In an exemplary embodiment, the candidate web page information for the next action may be the element information related to the next action among the information of each element in the current web page, which is crucial for the selection and execution of the subsequent action.

[0095] In an exemplary embodiment, the candidate web page information for the next action may include information of multiple elements in the current web page.

[0096] In an exemplary embodiment, the candidate web page information for the next action is content processed by a hypertext simplification algorithm, which retains the basic structure of HTML and important content information and eliminates redundant or destructive elements that may hinder model understanding.

[0097] In some embodiments, step 202 includes: extracting keywords for the next action from the description of the next action, retrieving keywords for the next action from information of each element in the current web page, and determining element information that matches the keywords for the next action as candidate web page information for the next action.

[0098] In other embodiments, Figure 4 As shown, step 202 includes:

[0099] Step 401, determining key information of each element in the current web page.

[0100] A single web page interface contains thousands of elements at least, and tens of thousands of elements at most. The levels and structures of each element are complex and diverse, and contain a lot of redundant information. Therefore, it is difficult to directly search for element information and the error rate is high. Determining the key information of each element in the current web page can quickly adjust the structure of the HTML information of the current web page and simplify the content information.

[0101] In an exemplary embodiment, the key information includes Tag and Xpath (Xml Path Language, XML Path Language, Extensible Markup Language Path Language).

[0102] In an exemplary embodiment, a tag may be a natural language or a character string.

[0103] In some embodiments, step 401 may include: 1) preprocessing a specific query string and adding a specific tag to identify the query content; 2) initializing the HTML content and related attributes of the current web page, including a whitelist and a blacklist, and determining the HTML tags and attributes that need to be retained or deleted; 3) recursively cleaning up useless attributes in HTML elements, retaining the attributes in the whitelist, parsing HTML, extracting tags without sub-elements and generating string representations and XPaths of the tags, and integrating them into a strongly structured web page element library, which stores string representations and XPaths of the tags.

[0104] For example, if the element is a search button, the tag of the element includes semantic information of the word "search", location information of the search button, etc. A tag without child elements means that the tag cannot be decomposed into multiple tags.

[0105] Cleaning and structuring the information of each element in the current web page to obtain the key information of each element in the current web page can improve the efficiency of determining the candidate web page information for the next action and improve the efficiency and experience of web page interaction.

[0106] Step 402: Generate a first vector corresponding to each key information of each element.

[0107] Step 403, generating a second vector describing the next step of action.

[0108] Step 404 : based on each of the first vectors and the second vectors, filter out the key information of the target element from the key information of each element.

[0109] In an exemplary embodiment, steps 402, 403 and 404 may be to input key information of each element and a description of the next action into a semantic vector model (BAAI General Embedding, BGE) to obtain key information of the target element.

[0110] In an exemplary embodiment, step 404 may be to determine the similarity between the first vector and the second vector based on each of the first vector and the second vector, and to filter out the key information of the target element from the key information of each element based on each of the similarities.

[0111] In an exemplary embodiment, the key information of the target element is filtered out from the key information of each element based on each similarity. The similarities may be sorted from high to low, and the key information of the elements corresponding to the first N similarities may be determined as the key information of the target element; or the key information of the elements corresponding to the similarities greater than the similarity threshold may be determined as the key information of the target element. For example, the key information of the six elements most relevant to the next action description may be determined as the key information of the target element.

[0112] Step 405: determine the key information of the target element as candidate web page information for the next action.

[0113] Convert the key information of each element in the current web page and the description of the next action into a semantic vector to achieve efficient semantic matching and accurate retrieval, reduce the time to find the target element, improve the efficiency of determining the candidate web page information for the next action, and improve the efficiency and experience of web page interaction.

[0114] Step 203, based on the task to be executed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, the screenshot of the previous action, the description of the next action and the candidate webpage information of the next action, determine the webpage operation information of the next action.

[0115] First, a next action description is generated, and then based on the next action description, candidate web page information for the next action is determined, thereby improving the efficiency of screening out candidate web page information, and then determining web page operation information for the next action, thereby further improving the efficiency of determining web page operation information for the next action.

[0116] In some embodiments, step 203 includes: extracting image semantic information of the current action from a screenshot of the current action; extracting image semantic information of the previous action from a screenshot of the previous action; determining the web page operation information of the next action based on the task to be performed, the web page operation information of the current action, the image semantic information of the current action, the web page operation information of the previous action, the image semantic information of the previous action, the description of the next action, and the candidate web page information of the next action.

[0117] In an exemplary embodiment, the image semantic information of the current action may be the information in the screenshot of the current action expressed in natural language, and the image semantic information of the previous action may be the image semantic information of the previous action expressed in natural language. Based on the task to be performed, the web page operation information of the current action, the image semantic information of the current action, the web page operation information of the previous action, the image semantic information of the previous action, the description of the next action, and the candidate web page information of the next action, the web page operation information of the next action is determined. The web page operation information of the next action may be obtained through a large language model based on the task to be performed, the web page operation information of the current action, the image semantic information of the current action, the web page operation information of the previous action, the image semantic information of the previous action, the description of the next action, and the candidate web page information of the next action.

[0118] In other embodiments, Figure 5 As shown, step 203 includes:

[0119] Step 501, generating a second prompt word based on the task to be executed, web page operation information of the current action, a screenshot of the current action, web page operation information of the previous action, a screenshot of the previous action, a description of the next action, and candidate web page information of the next action.

[0120] Step 502: input the second prompt word into the operation information generation model to obtain the web page operation information of the next step.

[0121] Among them, the second prompt word is used to instruct the operation information generation model to generate web page operation information for the next action based on the task to be executed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action, the screenshot of the previous action, the description of the next action and the candidate web page information of the next action.

[0122] In an exemplary embodiment, the operation information generation big model can be a multimodal big model. For example, the big model is an open source multimodal big model LLaVA-NeXT, which supports multi-image input and demonstrates powerful capabilities in image reasoning, video analysis, 3D scene understanding, and complex visual command tasks.

[0123] By generating a large model through operation information, the user needs are perceived from the tasks to be performed, and the association between the text information and image semantic information in the current action, the association between the text information and image semantic information in the previous action, and the association between the current action and the previous action are fully understood. In combination with the description of the next action and the candidate web page information of the next action, the web page operation information of the next action can be accurately given. When faced with a complex and diverse dynamic web page environment, the next action can be accurately judged, and the web page information corresponding to the action can be accurately selected from the candidate web page information of the next action, and then the web page operation information of the next action can be accurately given, thereby improving the accuracy of the web page operation information of the next action.

[0124] In some embodiments, Figure 6 As shown, after step 203, the web page action execution method further includes:

[0125] Step 601, generating evaluation prompt words based on the task to be executed, the webpage operation information of the current action, the webpage operation information of the previous action, the next action description, the candidate webpage information of the next action and the webpage operation information of the next action.

[0126] Step 602, input the evaluation prompt words into the evaluation model to obtain the evaluation result corresponding to the web page operation information of the next action.

[0127] Among them, the evaluation prompt words are used to instruct the evaluation model to generate an evaluation result corresponding to the web page operation information of the next action based on the task to be executed, the web page operation information of the current action, the web page operation information of the previous action, the description of the next action, the candidate web page information of the next action, and the web page operation information of the next action.

[0128] Step 203 has a strong dynamic web page perception capability by virtue of the multimodal large model, and can generate accurate web page operation information for the next action. However, actual web pages are complex and diverse, and the same web page may have semantically similar elements, and the ability of step 202 to generate candidate web page information for the next action is limited, so there are still cases where the web page operation information for the next action is predicted incorrectly. By evaluating the large model to obtain the evaluation result corresponding to the web page operation information for the next action, it is possible to determine whether the web page operation information for the next action is correct, thereby improving the accuracy of the execution of the next action.

[0129] The evaluation model can point out whether there is a problem with the webpage operation information of the next action and provide specific problem points. If there is no problem, step 102 is currently executed: based on the webpage operation information of the next action, the next action is executed; if there is a problem, the webpage operation information of the next action with a low score is corrected. The whole process can improve the accuracy of action execution and the reliability of results through continuous reflection and correction, thereby improving the accuracy of action execution.

[0130] In an exemplary embodiment, the evaluation large model may be a large language model. For example, the evaluation large model is an open source large language model LLaMA3.

[0131] In the exemplary embodiment, the evaluation result includes the content worthy of recognition, the areas that need improvement and the reply score. As required, the evaluation result may also include other content, which is not limited by the present application.

[0132] In an exemplary embodiment, the evaluation model evaluates the webpage operation information of the next action based on the reflection ability of the large language model, mainly from the three dimensions of accuracy, reliability, and interpretability. For example, the webpage operation information of the next action includes the next action and the webpage information corresponding to the next action, and the specific scoring criteria for the reply score can be: 0 points (the webpage information corresponding to the next action is not in the candidate webpage information of the next action), 0.5 points (the next action is wrong or does not match the description of the next action), 1 point (the next action is correct, and the webpage information corresponding to the next action is in the candidate webpage information of the next action and is correct).

[0133] In the exemplary embodiment, the evaluation prompt words are as follows:

[0134] “You are an automated web navigation assistant. Your task is to score the responses based on the task to be performed..., the web operation information of the current action..., the web operation information of the previous action..., the description of the next action..., the candidate web page information of the next action... and the web operation information of the next action..., combined with the given knowledge and areas that need improvement...”

[0135] In an exemplary embodiment, the task to be executed is "help me search for a mobile phone of brand E, 256G memory, and then a folding screen on application A", the web page operation information of the previous action is the HTML corresponding to "mouse click on the search button", the web page operation information of the current action is the HTML corresponding to "type brand E mobile phone in the search box", the next action is described as "find 256G body memory information", the candidate web page information of the next action is the HTML corresponding to the key information of multiple elements of the current web page, the web page operation information of the next action is the HTML corresponding to "mouse click on element X", and the evaluation result corresponding to the web page operation information of the next action output by the evaluation large model is as follows: Content worthy of recognition: {"The web page information corresponding to the next action is in the candidate web page information for the next action"}; Areas that need improvement: {"The result does not match the description of the next action": "The current web page types brand E mobile phone, and clicks to search for 256G memory. The large model chooses to click element X, which is an error"}; Response score: {"0.5 points": "The result does not match the description of the next action"}.

[0136] In some embodiments, Figure 7 As shown, after step 602, the web page action execution method further includes:

[0137] Step 701, when the evaluation result corresponding to the web page operation information of the next action indicates that the web page operation information of the next action is wrong, generate error correction prompt words based on the task to be executed, the web page operation information of the current action, the web page operation information of the previous action, the next action description, the candidate web page information of the next action, the web page operation information of the next action, and the evaluation result corresponding to the web page operation information of the next action.

[0138] Step 702, input the error correction prompt word into the error correction model to obtain the corrected web page operation information for the next action.

[0139] Among them, the error correction prompt words are used to instruct the error correction model to generate corrected web page operation information for the next action based on the task to be executed, the web page operation information of the current action, the web page operation information of the previous action, the description of the next action, the candidate web page information of the next action, the web page operation information of the next action, and the evaluation results corresponding to the web page operation information of the next action.

[0140] By correcting the web page operation information of the incorrectly predicted next action, the accuracy of the execution of each operation step can be effectively improved. The entire process can improve the accuracy of action execution and the reliability of results through continuous reflection and error correction, thereby improving the accuracy of action execution.

[0141] In an exemplary embodiment, the error correction large model may be a large language model. For example, the error correction large model is an open source large language model LLaMA3.

[0142] In an exemplary embodiment, the error correction model mainly corrects the web page operation information of the next action corresponding to the "reply score" less than 1.

[0143] In an exemplary embodiment, the error correction prompt words are as follows:

[0144] "You are an automated web navigation assistant. Your task is based on the task to be performed..., the web operation information of the current action..., the web operation information of the previous action..., the description of the next action..., the candidate web information of the next action... and the web operation information of the next action... On the premise of retaining the good content of the web operation information of the next action, based on the content worthy of recognition..., the areas that need improvement... and the response score..., correct the web operation information of the next action."

[0145] In some embodiments, after step 702, the web page action execution method further includes:

[0146] The webpage operation information of the next action is replaced with the corrected webpage operation information of the next action, and the process returns to step 601 until the evaluation result indicates that the webpage operation information is correct.

[0147] By repeatedly evaluating and correcting the web page operation information of the incorrectly predicted next action, the accuracy of the execution of each operation step can be effectively improved. The entire process can improve the accuracy of action execution and the reliability of results through continuous reflection and error correction, thereby improving the accuracy of action execution.

[0148] In the exemplary embodiment, the whole process is performed for at most two cycles, i.e., "scoring-correction-scoring-correction-scoring". If the "response score" in the middle satisfies 1 point, the loop is jumped out and step 102 is executed: based on the webpage operation information of the next action, the next action is executed. This reduces the situation where the execution efficiency of the next action is low due to too many cycles, which reduces the efficiency and experience of webpage interaction.

[0149] Step 102, executing the next action based on the webpage operation information of the next action.

[0150] In some embodiments, the webpage operation information includes an action and webpage information corresponding to the action. The webpage operation information of the next action includes the next action and webpage information corresponding to the next action.

[0151] In some embodiments, the action includes at least one of a mouse click (Click), a mouse hover (Hover), an element selection (Select), typing content (Type), scrolling up and down a web page (Scroll_web), moving forward and backward a web page (Go), jumping a web page (Jump_web), and generating a task conclusion (Finish).

[0152] These actions simulate the actual operations of users, and basically meet the needs of most operations on web pages.

[0153] In the exemplary embodiment, as shown in Table 1, it is a schematic diagram of actions and action descriptions.

[0154] Table 1

[0155] Action Action Description Click{element description} Click on an element Hover{element description} Hover the mouse over an element Select{Description of the selected element} Select an element Type{Type the content in the relevant element} Type in an element Scroll_web{up / down} Scroll up or down the web page Go{forward / backward} Go forward or back on a web page Jump_web{URL of the web page to be redirected} Web page jump Finish{Generate relevant reply words} Complete the process operation prompt

[0156] In some embodiments, step 102 includes: extracting an element path corresponding to the next action from the web page information corresponding to the next action, and executing the next action on the element path corresponding to the next action.

[0157] In an exemplary embodiment, the web page information includes tags and Xpath, where Xpath is an element path.

[0158] In other embodiments, step 102 includes: obtaining an action function corresponding to the next action; and executing the action function corresponding to the next action based on the web page information corresponding to the next action to implement the execution of the next action.

[0159] The execution steps corresponding to the action are encapsulated into an action function, which is used to execute the action in a real-time environment, thus realizing automatic web program operation.

[0160] In an exemplary embodiment, as shown in Table 2, actions and action functions are shown.

[0161] Table 2

[0162]

[0163]

[0164] In an exemplary embodiment, some parameters in the action function, for example, the parameter Content in Type{type content in the relevant element}: the content to be entered, and the parameter URL in Jump_web{URL of the web page to be jumped to}: the URL of the web page to be jumped to, can be extracted from the next action description.

[0165] In summary, in the present application, based on the task to be executed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action and the screenshot of the previous action, the web page operation information of the next action is determined, the user needs are perceived from the task to be executed, and based on the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action and the screenshot of the previous action, the association between the text information and the image semantic information within the current action, the association between the text information and the image semantic information within the previous action, and the association between the current action and the previous action are fully understood, the changes in the web page are fully learned, and then the web page operation information of the next action is determined, which conforms to the logic of human operation, is accurate and reliable, and improves the accuracy of the web page operation information of the next action. Based on the web page operation information of the next action, the next action is executed, and on the basis of improving the accuracy of the web page operation information of the next action, the accuracy of the next action to be executed is further improved, thereby improving the accuracy of the execution of the web page action.

[0166] Exemplary Devices

[0167] Correspondingly, the embodiment of the present application also provides a web page action execution device, such as Figure 8 As shown, the web page action execution device includes:

[0168] The processing unit 801 is used to determine the webpage operation information of the next action based on the task to be executed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action and the screenshot of the previous action;

[0169] The execution unit 802 is used to execute the next action based on the webpage operation information of the next action.

[0170] Optionally, the processing unit 801 includes:

[0171] A first processing subunit, configured to determine a next action description based on the task to be executed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, and the screenshot of the previous action;

[0172] A second processing subunit is used to determine candidate web page information for the next action based on the next action description;

[0173] The third processing sub-unit is used to determine the web page operation information of the next action based on the task to be executed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action, the screenshot of the previous action, the next action description and the candidate web page information of the next action.

[0174] Optionally, the first processing subunit is specifically configured to:

[0175] Generate a first prompt word based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, and the screenshot of the previous action;

[0176] Input the first prompt word into the action description generation model to obtain the next action description;

[0177] Among them, the first prompt word is used to instruct the action description generation model to generate the next action description based on the task to be performed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action and the screenshot of the previous action.

[0178] Optionally, the second processing subunit is specifically configured to:

[0179] Determine the key information of each element in the current web page;

[0180] Generate a first vector corresponding to each key information of each element;

[0181] Generate a second vector describing the next step action;

[0182] Based on each of the first vectors and the second vectors, filtering out key information of a target element from key information of each of the elements;

[0183] The key information of the target element is determined as candidate web page information for the next action.

[0184] Optionally, the third processing subunit is specifically configured to:

[0185] Generate a second prompt word based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, the screenshot of the previous action, the next action description, and the candidate webpage information of the next action;

[0186] Inputting the second prompt word into the operation information generation model to obtain the webpage operation information of the next step action;

[0187] Among them, the second prompt word is used to instruct the operation information generation model to generate the web page operation information of the next action based on the task to be executed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action, the screenshot of the previous action, the next action description and the candidate web page information of the next action.

[0188] Optionally, the webpage operation information includes an action and webpage information corresponding to the action;

[0189] The action includes at least one of mouse clicking, mouse hovering, element selection, content typing, web page scrolling up and down, web page forward and backward, web page jumping, and generating task conclusion.

[0190] Optionally, the execution unit 802 is specifically configured to:

[0191] Obtaining the action function corresponding to the next step action;

[0192] Based on the web page information corresponding to the next action, the action function corresponding to the next action is executed to implement the next action.

[0193] Optionally, the web page action execution device further includes:

[0194] An evaluation unit, configured to generate an evaluation prompt word based on the task to be executed, the webpage operation information of the current action, the webpage operation information of the previous action, the next action description, the candidate webpage information of the next action, and the webpage operation information of the next action; input the evaluation prompt word into the evaluation big model to obtain an evaluation result corresponding to the webpage operation information of the next action;

[0195] Among them, the evaluation prompt word is used to indicate that the evaluation model generates an evaluation result corresponding to the web page operation information of the next action based on the task to be executed, the web page operation information of the current action, the web page operation information of the previous action, the next action description, the candidate web page information of the next action and the web page operation information of the next action.

[0196] Optionally, the web page action execution device further includes:

[0197] The error correction unit is used to generate an error correction prompt word based on the task to be executed, the webpage operation information of the current action, the webpage operation information of the previous action, the next action description, the candidate webpage information of the next action, the webpage operation information of the next action, and the evaluation result corresponding to the webpage operation information of the next action when the evaluation result corresponding to the webpage operation information of the next action indicates that the webpage operation information of the next action is wrong; input the error correction prompt word into the error correction big model to obtain the corrected webpage operation information of the next action;

[0198] Among them, the error correction prompt word is used to instruct the error correction model to generate corrected web page operation information for the next action based on the task to be executed, the web page operation information of the current action, the web page operation information of the previous action, the next action description, the candidate web page information of the next action, the web page operation information of the next action, and the evaluation result corresponding to the web page operation information of the next action.

[0199] Optionally, the web page action execution device further includes:

[0200] A replacement unit is used to replace the web page operation information of the next action with the corrected web page operation information of the next action, and return to the step of generating evaluation prompt words based on the task to be executed, the web page operation information of the current action, the web page operation information of the previous action, the next action description, the candidate web page information of the next action and the web page operation information of the next action, until the evaluation result indicates that the web page operation information is correct.

[0201] The web page action execution device provided in this embodiment belongs to the same application concept as the web page action execution method provided in the above embodiments of this application, and can execute the web page action execution method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the web page action execution method. For technical details not fully described in this embodiment, please refer to the specific processing content of the web page action execution method provided in the above embodiments of this application, and will not be repeated here.

[0202] The functions implemented by the above processing unit 801 and execution unit 802 may be implemented by the same or different processors, respectively, and the embodiment of the present application is not limited thereto.

[0203] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0204] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0205] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0206] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0207] Exemplary Electronic Devices

[0208] The present application embodiment provides an electronic device, see Fig. 9 As shown, the device includes:

[0209] Memory 200 and processor 210;

[0210] The memory 200 is connected to the processor 210 and is used to store programs;

[0211] The processor 210 is used to implement the web page action execution method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0212] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0213] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0214] A bus may include a pathway that transfers information between components of a computer system.

[0215] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0216] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0217] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0218] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0219] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0220] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0221] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any web page action execution method provided in the above embodiments of the present application.

[0222] Exemplary computer program products and storage media

[0223] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the web page action execution method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0224] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0225] In addition, the embodiment of the present application may also be a storage medium on which a computer program is stored. The computer program is executed by a processor to execute the steps of the web page action execution method according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically the following steps may be implemented:

[0226] Step 101, determining web page operation information for a next action based on the task to be performed, web page operation information for a current action, a screenshot of the current action, web page operation information for a previous action, and a screenshot of the previous action;

[0227] Step 102, executing the next action based on the webpage operation information of the next action.

[0228] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0229] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0230] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0231] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0232] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0233] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0234] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0235] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0236] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0237] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0238] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A web page action execution method, characterized in that: include: Determine web page operation information for a next action based on the task to be performed, web page operation information for a current action, a screenshot of the current action, web page operation information for a previous action, and a screenshot of the previous action; Based on the webpage operation information of the next action, the next action is executed.

2. The web page action execution method according to claim 1, characterized in that: The determining of the webpage operation information of the next action based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action and the screenshot of the previous action includes: Determine a next action description based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, and the screenshot of the previous action; Based on the next action description, determine candidate web page information for the next action; Based on the task to be executed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action, the screenshot of the previous action, the next action description and the candidate web page information of the next action, the web page operation information of the next action is determined.

3. The web page action execution method according to claim 2, characterized in that: The determining of the next action description based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, and the screenshot of the previous action includes: Generate a first prompt word based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, and the screenshot of the previous action; Input the first prompt word into the action description generation model to obtain the next action description; Among them, the first prompt word is used to instruct the action description generation model to generate the next action description based on the task to be performed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action and the screenshot of the previous action.

4. The web page action execution method according to claim 2, characterized in that: The step of determining candidate web page information for the next action based on the next action description includes: Determine the key information of each element in the current web page; Generate a first vector corresponding to each key information of each element; Generate a second vector describing the next step action; Based on each of the first vectors and the second vectors, filtering out key information of a target element from key information of each of the elements; The key information of the target element is determined as candidate web page information for the next action.

5. The web page action execution method according to claim 2, characterized in that: The determining the webpage operation information of the next action based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, the screenshot of the previous action, the next action description and the candidate webpage information of the next action includes: Generate a second prompt word based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, the screenshot of the previous action, the next action description, and the candidate webpage information of the next action; Inputting the second prompt word into the operation information generation model to obtain the webpage operation information of the next step action; Among them, the second prompt word is used to instruct the operation information generation model to generate the web page operation information of the next action based on the task to be executed, the web page operation information of the current action, the screenshot of the current action, the web page operation information of the previous action, the screenshot of the previous action, the next action description and the candidate web page information of the next action.

6. The web page action execution method according to claim 1, characterized in that: The webpage operation information includes actions and webpage information corresponding to the actions; The action includes at least one of mouse clicking, mouse hovering, element selection, content typing, web page scrolling up and down, web page forward and backward, web page jumping, and generating task conclusion.

7. The web page action execution method according to claim 6, characterized in that: The performing of the next action based on the webpage operation information of the next action includes: Obtaining the action function corresponding to the next step action; Based on the web page information corresponding to the next action, the action function corresponding to the next action is executed to implement the next action.

8. The web page action execution method according to claim 2, characterized in that: After determining the webpage operation information of the next action based on the task to be performed, the webpage operation information of the current action, the screenshot of the current action, the webpage operation information of the previous action, the screenshot of the previous action, the next action description, and the candidate webpage information of the next action, the method further includes: Generate an evaluation prompt word based on the task to be performed, the webpage operation information of the current action, the webpage operation information of the previous action, the next action description, the candidate webpage information of the next action, and the webpage operation information of the next action; Input the evaluation prompt word into the evaluation big model to obtain the evaluation result corresponding to the webpage operation information of the next action; Among them, the evaluation prompt word is used to indicate that the evaluation model generates an evaluation result corresponding to the web page operation information of the next action based on the task to be executed, the web page operation information of the current action, the web page operation information of the previous action, the next action description, the candidate web page information of the next action and the web page operation information of the next action.

9. The web page action execution method according to claim 8, characterized in that: After inputting the evaluation prompt word into the evaluation macro model to obtain the evaluation result corresponding to the webpage operation information of the next action, the method further includes: In the case where the evaluation result corresponding to the webpage operation information of the next action indicates that the webpage operation information of the next action is wrong, generating an error correction prompt word based on the task to be executed, the webpage operation information of the current action, the webpage operation information of the previous action, the next action description, the candidate webpage information of the next action, the webpage operation information of the next action, and the evaluation result corresponding to the webpage operation information of the next action; Input the error correction prompt word into the error correction large model to obtain the corrected web page operation information for the next action; Among them, the error correction prompt word is used to instruct the error correction model to generate corrected web page operation information for the next action based on the task to be executed, the web page operation information of the current action, the web page operation information of the previous action, the next action description, the candidate web page information of the next action, the web page operation information of the next action, and the evaluation result corresponding to the web page operation information of the next action.

10. The web page action execution method according to claim 9, characterized in that: After inputting the error correction prompt word into the error correction large model to obtain the corrected webpage operation information for the next action, the method further includes: The web page operation information of the next action is replaced with the corrected web page operation information of the next action, and the step of generating evaluation prompt words based on the task to be executed, the web page operation information of the current action, the web page operation information of the previous action, the next action description, the candidate web page information of the next action and the web page operation information of the next action is returned to, until the evaluation result indicates that the web page operation information is correct.

11. A web page action execution device, characterized in that: include: A processing unit, configured to determine web page operation information for a next action based on the task to be performed, web page operation information for a current action, a screenshot of the current action, web page operation information for a previous action, and a screenshot of the previous action; The execution unit is used to execute the next action based on the webpage operation information of the next action.

12. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the web page action execution method as described in any one of claims 1 to 10 by running the program in the memory.

13. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to execute the web page action execution method as claimed in any one of claims 1 to 10.

Images