Page operation method and device based on large language model

Abstract

The application provides a page operation method and device based on a large language model, wherein the page operation method based on the large language model comprises the following steps: receiving page operation demand information for a target business system; acquiring a current business page of the target business system, and generating a page operation text according to the current business page and the page operation demand information, wherein the page operation text comprises historical operation information for the target business system; inputting the page operation text into a page analysis large model, and acquiring current page description information and current operation information output by the page analysis large model; in the case that the current page description information does not satisfy a page operation termination condition, processing the current business page according to the current operation information, and updating the historical operation information according to the current operation information.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a page manipulation method based on a large language model. This application also relates to a page manipulation apparatus based on a large language model, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Page operation automation technology refers to the technology of automatically manipulating relevant pages through scripts. It is widely used in scenarios such as information retrieval and page testing.

[0003] Page automation technology requires pre-recording or developing corresponding scripts based on different operation objectives. Since the requirements of each task differ, separate scripts need to be created for each task, which is time-consuming and laborious. Furthermore, each step of the operation based on the script is executed according to pre-defined page, button, menu, input box, and other control information. If the page's interaction method changes (such as the position of buttons, menus, or input boxes being adjusted), or if a temporary error occurs on the page (such as an error pop-up or temporary confirmation message), the script cannot continue to perform the page automation task, and the script needs to be updated. Summary of the Invention

[0004] In view of this, embodiments of this application provide a page manipulation method based on a large language model. This application also relates to a page manipulation apparatus based on a large language model, a computing device, a computer-readable storage medium, and a computer program product, to solve the aforementioned problems existing in the prior art.

[0005] According to a first aspect of the embodiments of this application, a page operation method based on a large language model is provided, including:

[0006] Receive page operation request information for the target business system;

[0007] Obtain the current business page of the target business system, and generate page operation text based on the current business page and the page operation requirement information, wherein the page operation text includes historical operation information for the target business system;

[0008] Input the page operation text into the page parsing big model, and obtain the current page description information and current operation information output by the page parsing big model;

[0009] If the current page description information does not meet the page operation termination condition, the current business page is processed according to the current operation information, and the historical operation information is updated according to the current operation information.

[0010] According to a second aspect of the embodiments of the present application, a page operation device based on a large language model is provided, comprising:

[0011] a receiving module configured to receive page operation demand information for a target business system;

[0012] a generating module configured to obtain a current business page of the target business system, and generate a page operation text according to the current business page and the page operation demand information, wherein the page operation text comprises historical operation information for the target business system;

[0013] an obtaining module configured to input the page operation text into a page parsing large model, and obtain current page description information and current operation information output by the page parsing large model;

[0014] an operation module configured to, in a case where the current page description information does not satisfy a page operation termination condition, process the current business page according to the current operation information, and update the historical operation information according to the current operation information.

[0015] According to a third aspect of the embodiments of the present application, a computing device is provided, comprising:

[0016] a memory and a processor;

[0017] the memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, and the computer programs / instructions, when executed by the processor, implement the steps of the above-mentioned page operation method based on a large language model.

[0018] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer programs / instructions, and the computer programs / instructions, when executed by a processor, implement the steps of the above-mentioned page operation method based on a large language model.

[0019] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, comprising computer programs / instructions, and the computer programs / instructions, when executed by a processor, implement the steps of the above-mentioned page operation method based on a large language model.

[0020] The page operation method based on a large language model provided in the application receives page operation demand information for a target business system; acquires a current business page of the target business system, generates a page operation text according to the current business page and the page operation demand information, wherein the page operation text includes historical operation information for the target business system; inputs the page operation text into a page analysis large model, acquires current page description information and current operation information output by the page analysis large model; in the case where the current page description information does not satisfy a page operation termination condition, processes the current business page according to the current operation information, and updates the historical operation information according to the current operation information.

[0021] An embodiment of the application realizes generation of a page operation text according to a current business page of a target business system, page operation demand information and historical operation information, and input of the page operation text into a page analysis large model. The page analysis large model can analyze the current business page, the page operation demand information and the historical operation information, select a reasonable operation in the current business page by using the understanding ability and planning ability of the large model, and perform the next operation according to the interaction return of the target business system. The problem of interruption of automatic operation caused by an accidental page in the page automatic operation task is solved. The adaptability of the page automatic operation task is improved, and the processing efficiency of the page automatic operation is improved.

[0022] Secondly, the historical operation information informs the page analysis large model of the steps completed before, and limits the page analysis large model to focus on the current step without predicting the subsequent operation result, so that the page analysis large model focuses on processing the current step and understanding the progress of the task, and the accuracy of the page analysis large model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a flowchart of a page operation method based on a large language model provided by an embodiment of the application;

[0024] Figure 2 is a processing flowchart of a page operation method based on a large language model applied to automatic downloading of files provided by an embodiment of the application;

[0025] Figure 3 is a structural schematic diagram of a page operation device based on a large language model provided by an embodiment of the application;

[0026] Figure 4 is a structural block diagram of a computing device provided by an embodiment of the application. DETAILED DESCRIPTION

[0027] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details. In other instances, well-known methods have not been described in detail in order not to unnecessarily obscure aspects of the present application.

[0028] The terminology used in this description is for the purpose of describing particular embodiments only and is not intended to limit one or more embodiments of the present application. As used in one or more embodiments of the present application and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0029] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another only. For example, without departing from the scope of one or more embodiments, first can be termed second, and similarly, second can be termed first. The term "if' as used herein, depending on the context, can be interpreted to mean "when" or "in response to determining" or "in response to a determination."

[0030] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards in the relevant region, and provide corresponding operation portal for user to choose authorization or refusal.

[0031] In one or more embodiments of the present specification, a large model refers to a deep learning model with a large number of model parameters, usually containing hundreds of millions, tens of billions, hundreds of billions, tens of billions or even more than one hundred billion model parameters. The large model can also be called a foundation model. Through large-scale unlabeled corpus pre-training of the large model, a pre-training model with more than one hundred million parameters is output. Such a model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large language model (LLM) and multi-modal pre-training model.

[0032] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0033] First, the terms and concepts involved in one or more embodiments of this application will be explained.

[0034] Large Language Models (LLMs): In machine learning and artificial intelligence, large language models typically refer to deep learning models with a large number of parameters and complex structures. Due to their massive scale, these models can learn and store vast amounts of knowledge, possessing excellent generalization capabilities and the ability to solve complex problems. Large models can perform tasks in various scenarios, including multi-task, multi-domain, and multimodal environments, and are widely used in fields such as natural language processing, image recognition, and speech recognition.

[0035] In real-world scenarios such as automated testing of page functionality and information retrieval, page operation automation technology is essential. Page operation automation technology refers to the technique of automatically performing operations on relevant pages using scripts.

[0036] However, current page operation automation technologies have several limiting problems. First, depending on the purpose of the operation, relevant scripts need to be pre-recorded or developed. In each operation within the script, the positions (x, y coordinates) or IDs of controls such as buttons, menus, and input boxes must be pre-defined to help the automation tool locate and simulate clicks and input. The script recording or development process is time-consuming and labor-intensive. Second, if the page's interaction method changes (such as adjusting the positions of buttons or menu controls), or if temporary errors occur on the page (such as error pop-ups or message pop-ups), the script will not know how to handle the new interface, requiring manual intervention from users to update accordingly, thus failing to effectively automate the process.

[0037] This application provides a page operation method based on a large language model. This application also relates to a page operation device based on a large language model, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0038] Figure 1 The flowchart illustrates a page operation method based on a large language model according to an embodiment of this application, which specifically includes the following steps:

[0039] Step 102: Receive page operation request information for the target business system.

[0040] The target business system can be understood as the business system that requires the application of page operation automation technology. For example, the target business system could be an automated testing system or an automated file download system. Users expect to use page operation automation technology to more conveniently perform corresponding business processing tasks within the target business system.

[0041] Page operation requirement information can be understood as the information that the user needs to operate on the target business system. Specifically, it can include what operation the user wants the target business system to perform, the parameter information required by the target business system to perform the relevant operation, and the function information that the target business system needs to call to perform the relevant operation.

[0042] Specifically, in one embodiment provided in this application, receiving page operation request information for the target business system includes:

[0043] Receive business purpose information, business parameter information, and operation function information for the target business system.

[0044] In practical applications, page operation requirements are composed of multiple pieces of information. Specifically, it can receive information about the user's business objective, business parameters, and operation functions for the target business system. These elements together constitute the page operation requirements.

[0045] In this context, business purpose information can be understood as what operation the user wants the target business system to perform. For example, business purpose information could be "download a document from the website" or "test a certain system." Through business purpose information, we can understand the user's purpose in using the target business system.

[0046] Business parameter information can be understood as the parameter information that the target business system needs to use during the relevant operation. For example, when the business purpose information is "to download a document from the website", the business parameter information can be information such as "website URL, username, password, document name or ID".

[0047] Operation function information can be understood as the utility functions that the target business system needs to use during relevant operations. Examples include functions for inputting information, clicking the mouse, and refreshing the page.

[0048] Upon receiving a page operation request for the target business system, it becomes clear what kind of operation the user expects to perform, as well as the parameters and functions used in the operation. This facilitates subsequent processing.

[0049] Step 104: Obtain the current business page of the target business system, and generate page operation text based on the current business page and the page operation requirement information, wherein the page operation text includes historical operation information for the target business system.

[0050] The target business system has multiple pages during application. The method provided in this application aims to understand the current page's status and perform subsequent processing based on its content. Therefore, it is necessary to obtain the current business page of the target business system. The current business page can be understood as the page currently displayed by the target business system. It should be noted that the current business page is specifically a screenshot of the page currently displayed by the target business system.

[0051] In one specific embodiment provided in this application, obtaining the current business page of the target business system includes:

[0052] Use a screenshot tool to capture the current business page of the target business system.

[0053] In practical applications, obtaining the current business page can be achieved using a screenshot tool. Specifically, the screenshot tool is invoked to capture the current business page of the target business system.

[0054] Once the current business page is obtained, page operation text can be generated based on the current business page and page operation requirements. This page operation text can be understood as a prompt subsequently input into the large-scale page parsing model. In the large AI model, the prompt's role is to provide the AI ​​model with contextual information about the input and the parameters input to the model. The prompt helps the large model better understand the intent of the input and respond accordingly, improving the interpretability and accessibility of the large model.

[0055] In the method provided in this application, after obtaining the current business page, the corresponding page operation text can be generated according to the current business page and page operation requirements, which can be used to help the large model of page parsing understand the current business page.

[0056] In one specific embodiment provided in this application, page operation text is generated based on the current business page and the page operation requirement information, including:

[0057] Obtain historical operation information and model expectation information for the target business system, wherein the model expectation information determines the return information of the model;

[0058] The page operation requirement information, the current business page, the historical operation information, and the model expectation information are combined to generate page operation text.

[0059] In practical applications, the target business system may involve many steps when performing related processes. For example, during the automated testing of a certain function, it is necessary to log in to the test system and access the menu of the test function. This includes "login with username and password - access the sub-menu through the menu - access the page of the function to be tested through the sub-menu - test a specific function on the page of the function to be tested". To facilitate the subsequent understanding of the larger model, it is necessary to obtain historical operation information of the target business system so that the larger model can understand which step the target business system is currently executing.

[0060] Model expectation information can be understood as the information that the user expects the model to return. In other words, the large language model can obtain relevant content based on the model expectation information and return it.

[0061] For example, the model's expected information could be "The model needs to answer: Question 1, Question 2, Question 3, Information 1, Information 2," where Question 1, Question 2, and Question 3 are the questions the model is expected to answer based on the current business page. Information 1 and Information 2 are the information the model outputs based on the input content. This expected information can limit the output of large models, preventing them from outputting content that users don't need.

[0062] Accordingly, after obtaining historical operation information and model expectation information, these two types of information can be combined to generate page operation text. Specifically, the page operation requirement information, the current business page, historical operation information, and model expectation information are concatenated to generate page operation text.

[0063] In a specific embodiment provided in this application, an automated testing system is used as an example for explanation. The system receives page operation request information for the automated testing system and obtains the current business page of the automated testing system. It also obtains historical operation information already performed on the automated testing system and user expectations for the model, generating page operation text including:

[0064] "You are an automated testing assistant;"

[0065] 1. Your task is to test feature A of the website;

[0066] 2. The information you need includes: "Website: ****; Username: Zhang San; Password: 123456; Test Purpose: ****";

[0067] 3. Available utility functions include: "Input Text Related Functions"; "Mouse Click Related Functions";

[0068] 4. The steps that have already been completed are: "Step 1, Step 2...";

[0069] 5. Current business page;

[0070] 6. You need to answer: "Question 1, Question 2, Question 3, Message 1, Message 2"

[0071] 7. Precautions.

[0072] In the page operation text, 1 can be interpreted as business purpose information, 2 as business parameter information, 3 as operation function information, 4 as historical operation information, 5 as the current business page, and 6 as model expectation information. 7 represents considerations for model processing, such as limiting the large model to process only the current input information without making assumptions or expectations about subsequent operations, preventing model illusion problems, and improving the processing efficiency of large models.

[0073] Step 106: Input the page operation text into the page parsing big model, and obtain the current page description information and current operation information output by the page parsing big model.

[0074] After obtaining the page operation text, it is input into the large-scale page parsing model for processing. In this embodiment, the large-scale page parsing model can be understood as a large language model (LLM). A large language model is a deep learning model trained on massive amounts of text data. It can generate natural language text, deeply understand the meaning of text, and handle various natural language tasks, such as text summarization, question answering, and translation. The large-scale page parsing model can understand the content of the page operation text and provide corresponding feedback information based on the model's expected information.

[0075] In the method provided in this application, the page parsing model parses and understands the current business page based on the input page operation text, and outputs current page description information and current operation information. The current page description information can be understood as the page description information output by the page analysis model after parsing the current business page; the current operation information can be understood as the operation information determined based on the parsing of the current business page.

[0076] For example, the page manipulation text generated in the above steps can be input into the page parsing model for processing to obtain the output of the page parsing model:

[0077] 1. I see the following information:

[0078] -****

[0079] -****

[0080] -****

[0081] -****

[0082] 2. The next step is:

[0083] -****

[0084] -****

[0085] -****

[0086] 3. Description of each operation step:

[0087] -****

[0088] -****

[0089] -****".

[0090] In the output, 1 represents the current page description, 2 represents the current operation information, and 3 represents the explanation of each step or content in the current operation information.

[0091] The page parsing model generates corresponding current page description information and current operation information based on the input page operation text. This facilitates subsequent determination of whether the target business system meets the user's actual needs based on the current page description information, while the current operation information guides the target business system on what operations to perform on the current business page.

[0092] In one specific embodiment provided in this application, the method further includes:

[0093] Establish a communication connection between the target business system and the large page parsing model.

[0094] The embodiments provided in this application address issues arising during page operation automation in the target business system by linking a large-scale page parsing model with the target business system. To reduce user operations, data exchange can be established between the target business system and the large-scale page parsing model, i.e., a communication connection can be established between them. Specifically, the current business page in the target business system can be sent to the large-scale page parsing model via a data interface call, or data from the large-scale page parsing model can be sent to the target business system via a data interface. This achieves interconnection between the target business system and the large-scale page parsing model, reducing frequent user operations between the two systems.

[0095] Step 108: If the current page description information does not meet the page operation termination condition, process the current business page according to the current operation information, and update the historical operation information according to the current operation information.

[0096] After obtaining the current page description information output by the large page parsing model, if the current page description information does not meet the page operation termination condition, the current business page can be further processed according to the current operation information. This allows the target business system to perform corresponding operations based on the current operation information and obtain a new current business page. This process can then be repeated to generate page operation text based on the current business page and page operation requirements.

[0097] At the same time, the current operation information is recorded in the historical operation information. By updating the historical operation information, the current operation information is recorded, and the page will be informed of which operations have been performed before the current business page in the parsing model.

[0098] In one specific embodiment provided in this application, the method further includes:

[0099] Determine whether the page operation termination condition is met based on the current page description information and the page operation requirement information.

[0100] If the page operation termination conditions are met, processing of the target business system will be stopped.

[0101] In practical applications, after obtaining the current page description information, it is necessary to determine whether it meets the user's actual business needs. Specifically, this involves determining whether the page operation termination conditions are met based on the current page description information and the page operation requirement information.

[0102] If the page operation termination condition is met, the automated operation task for that page is complete, and processing for the target business system can stop. If the page termination condition has not yet been met, the subsequent operation of generating page operation text based on the current business page and page operation requirements information will continue.

[0103] Page operation termination conditions can be understood as the conditions under which an automated page operation task, determined based on the page operation requirement information, meets the technical requirements. For example, if the page operation requirement information for the target business system is "download a document from the website," and the current page description is "download completed," then the current page description information meets the page operation termination condition. As another example, if the page operation requirement information is "test a system," and the current page description is "test completed," then the current page description information meets the page operation termination condition.

[0104] That is, determining whether the page operation termination condition is met based on the current page description information and the page operation requirement information, including:

[0105] Obtain the business purpose information from the page operation requirement information;

[0106] Determine whether the current page description information meets the business objective information;

[0107] If so, then the page operation termination condition is met;

[0108] If not, then the page operation termination condition has not been met.

[0109] In practical applications, the page operation requirement information further includes business purpose information. This business purpose information is extracted from the page operation requirement information to determine whether the current page description satisfies the business purpose information. Specifically, the current page description information and the business purpose information can be input into a large language model for analysis to determine whether they match. If they match, it means the current page description information satisfies the business purpose information, and it can be further determined that the current page description information meets the page operation termination condition; otherwise, it is considered that the current page description information does not meet the page operation termination condition.

[0110] In one specific embodiment provided in this application, processing the current business page according to the current operation information includes:

[0111] Generate current page operation instructions based on the current operation information;

[0112] The current business page is processed based on the current page operation instructions, so that the target business system generates a new current business page.

[0113] The page parsing model outputs current operation information, meaning it provides the necessary actions for the current business page based on the input content. This current operation information can be understood as the planned actions to be performed on the current business page. During the processing of the current business page based on this current operation information, it's necessary to first generate current page operation instructions, which include the specific operation information for the current business page.

[0114] The current page operation command is sent to the target business system to perform operations on the current business page, thereby changing the current business page and obtaining a new current business page. The process continues to loop, retrieving the current business page from the target business system, generating page operation text based on the current business page and the page operation requirement information, inputting the page operation text into the page parsing model, and obtaining the current page description information and current operation information output by the page parsing model.

[0115] In one specific embodiment provided in this application, updating the historical operation information based on the current operation information includes:

[0116] Generate current operation description information based on the current operation information;

[0117] The historical operation information is updated based on the current operation description information, wherein the historical operation information records operation description information for the target business system.

[0118] Current operation information typically provides more detailed descriptions. Directly updating historical operation information with current operation information would result in information redundancy in the historical records. To simplify the content of historical operation information, corresponding current operation description information can be generated based on the current operation information. This current operation description information can be understood as a concise explanation of the current operation, and it is then updated in the historical operation information record. Specifically, the current operation description information is appended to the historical operation information record. The historical operation information record contains operation description information specific to the target business system.

[0119] The page operation method based on a large language model provided in this application includes receiving page operation request information for a target business system; obtaining the current business page of the target business system; generating page operation text based on the current business page and the page operation request information, wherein the page operation text includes historical operation information for the target business system; inputting the page operation text into a large page parsing model; obtaining current page description information and current operation information output by the large page parsing model; and, if the current page description information does not meet the page operation termination condition, processing the current business page according to the current operation information and updating the historical operation information according to the current operation information.

[0120] The method provided in this application generates page operation text based on the current business page, page operation requirements, and historical operation information of the target business system, and inputs this text into a large-scale page parsing model. This allows the large-scale page parsing model to analyze the current business page, page operation requirements, and historical operation information, leveraging its image understanding and planning capabilities to select appropriate operations on the current business page and proceed with the next operation based on the interaction feedback from the target business system. This solves the problem of automation interruption caused by occasional page occurrences in automated page operation tasks. It improves the adaptability of automated page operation tasks, thereby increasing the processing efficiency of automated page operations.

[0121] Secondly, by informing the page parsing model of the steps already completed through historical operation information, and limiting the page parsing model to focus on the current step without having to predict the results of subsequent operations, the page parsing model can concentrate on processing the current step and understand the progress of the task, thereby improving the accuracy of the page parsing model.

[0122] The following is in conjunction with the appendix Figure 2 Taking the application of the page manipulation method based on a large language model provided in this application in the automatic file download as an example, the page manipulation method based on a large language model will be further explained. Among them, Figure 2 This application provides a flowchart illustrating a page operation method based on a large language model for automatically downloading files, according to an embodiment of the present application. The method specifically includes the following steps:

[0123] Step 202: Receive the business purpose information, business parameter information, and operation function information for the file download system.

[0124] In the embodiments provided in this application, the received business purpose information is "download the patent publication document with application number 123 from website A". The business parameter information includes "website A's URL; username and password; application number: 123; file type: patent publication document". The operation function information includes "input text: function_input_text(label, text), where label is the title of the input control and text is the text to be entered; mouse click: function_mouse_click(x, y), where x represents the x-coordinate of the clicked position on the page and y represents the y-coordinate of the clicked position on the page, and both x and y are relative to the actual position on the page".

[0125] Step 204: Obtain the current business page of the file download system.

[0126] Step 206: Obtain historical operation information and model expectation information for the file download system.

[0127] In the embodiments provided in this application, the historical operation information is "Completed steps: 1. Enter username and password and log in; 2. Click the menu to enter the All Files page." The model's expected information is "What information did you see on the page? What do you need to do? Briefly describe the next step. Consider the utility functions you need to call. Return the utility functions and parameters to be called in JSON format, example: Instance 1."

[0128] Step 208: Combine page operation requirement information, current business page, historical operation information, and model expectation information to generate page operation text.

[0129] In the embodiments provided in this application, the information is concatenated to generate page operation text:

[0130] "You are a web page access task auto-execution assistant."

[0131] 1. Your task: Download the patent publication document with application number 123 from website A.

[0132] 2. The information you need: Website A's URL; username and password; application number: 123; document type: patent publication document.

[0133] 3. Available utility functions: Input text: function_input_text(label, text), where label is the title of the input control and text is the text to be entered; Mouse click: function_mouse_click(x, y), where x represents the x-coordinate of the clicked position on the page and y represents the y-coordinate of the clicked position on the page. Both x and y are relative to the actual position on the page.

[0134] 4. The steps already completed: "1. Enter username and password and log in; 2. Click the menu to go to the All Files page."

[0135] 5. Current business page.

[0136] 6. You need to answer:

[0137] What information did you see on the page?

[0138] What do you need to do? Briefly describe the next steps.

[0139] Consider the utility functions you need to call;

[0140] Return the utility functions and parameters to be called in JSON format. Example: Example 1.

[0141] 7. Note

[0142] The coordinates need to provide the actual position relative to the image size, not an assumed position;

[0143] You only need to provide the input or click action for this instance. The result will be sent to you next time. Do not assume or expect the page's processing result this time.

[0144] Step 210: Input the page operation text into the page parsing big model, and obtain the current page description information and current operation information output by the page parsing big model.

[0145] Step 212: Determine whether the description information on the current page meets the business objective information. If yes, proceed to step 214; otherwise, proceed to step 216.

[0146] Step 214: Stop processing for the file download system.

[0147] Step 216: Generate current page operation instructions based on the current operation information, process the current business page based on the current page operation instructions, and return to execute step 204.

[0148] The method provided in this application generates page operation text based on the current business page, page operation requirements, and historical operation information of the target business system, and inputs this text into a large-scale page parsing model. This allows the large-scale page parsing model to analyze the current business page, page operation requirements, and historical operation information, leveraging its understanding and planning capabilities to select appropriate operations on the current business page and proceed with the next operation based on the interaction feedback from the target business system. This solves the problem of automation interruption caused by occasional page occurrences in automated page operation tasks. It improves the adaptability of automated page operation tasks, thereby increasing the processing efficiency of automated page operations.

[0149] Secondly, by informing the page parsing model of the steps already completed through historical operation information, and limiting the page parsing model to focus on the current step without having to predict the results of subsequent operations, the page parsing model can concentrate on processing the current step and understand the progress of the task, thereby improving the accuracy of the page parsing model.

[0150] Corresponding to the above method embodiments, this application also provides an embodiment of a page operation device based on a large language model. Figure 3 This diagram illustrates the structure of a page operation device based on a large language model, according to an embodiment of this application. Figure 3 As shown, the device includes:

[0151] The receiving module 302 is configured to receive page operation request information for the target business system;

[0152] The generation module 304 is configured to obtain the current business page of the target business system and generate page operation text based on the current business page and the page operation requirement information, wherein the page operation text includes historical operation information for the target business system;

[0153] The acquisition module 306 is configured to input the page operation text into the page parsing big model and acquire the current page description information and current operation information output by the page parsing big model;

[0154] The operation module 308 is configured to process the current business page according to the current operation information and update the historical operation information according to the current operation information when the current page description information does not meet the page operation termination condition.

[0155] Optionally, the receiving module 302 is further configured to:

[0156] Receive business purpose information, business parameter information, and operation function information for the target business system.

[0157] Optionally, the generation module 304 is further configured to:

[0158] Use a screenshot tool to capture the current business page of the target business system.

[0159] Optionally, the generation module 304 is further configured to:

[0160] Obtain historical operation information and model expectation information for the target business system, wherein the model expectation information determines the return information of the model;

[0161] The page operation requirement information, the current business page, the historical operation information, and the model expectation information are combined to generate page operation text.

[0162] Optionally, the device further includes a determination module configured to:

[0163] Determine whether the page operation termination condition is met based on the current page description information and the page operation requirement information.

[0164] If the page operation termination conditions are met, processing of the target business system will be stopped.

[0165] Optionally, the determination module is further configured to:

[0166] Obtain the business purpose information from the page operation requirement information;

[0167] Determine whether the current page description information meets the business objective information;

[0168] If so, then the page operation termination condition is met;

[0169] If not, then the page operation termination condition has not been met.

[0170] Optionally, the operation module 308 is further configured to:

[0171] Generate current page operation instructions based on the current operation information;

[0172] The current business page is processed based on the current page operation instructions, so that the target business system generates a new current business page.

[0173] Optionally, the device further includes a connection establishment module configured to:

[0174] Establish a communication connection between the target business system and the large page parsing model.

[0175] Optionally, the operation module 308 is further configured to:

[0176] Generate current operation description information based on the current operation information;

[0177] The historical operation information is updated based on the current operation description information, wherein the historical operation information records operation description information for the target business system.

[0178] The apparatus provided in this application generates page operation text based on the current business page, page operation requirements, and historical operation information of the target business system, and inputs the page operation text into a large-scale page parsing model. This allows the large-scale page parsing model to analyze the current business page, page operation requirements, and historical operation information, leveraging its understanding and planning capabilities to select appropriate operations on the current business page and proceed with the next operation based on the interaction feedback from the target business system. This solves the problem of automation interruption caused by occasional page occurrences in automated page operation tasks. It improves the adaptability of automated page operation tasks, thereby increasing the processing efficiency of automated page operations.

[0179] Secondly, by informing the page parsing model of the steps already completed through historical operation information, and limiting the page parsing model to focus on the current step without having to predict the results of subsequent operations, the page parsing model can concentrate on processing the current step and understand the progress of the task, thereby improving the accuracy of the page parsing model.

[0180] The above is an illustrative scheme of a page operation device based on a large language model according to this embodiment. It should be noted that the technical solution of this page operation device based on a large language model and the technical solution of the page operation method based on a large language model described above belong to the same concept. For details not described in detail in the technical solution of the page operation device based on a large language model, please refer to the description of the technical solution of the page operation method based on a large language model described above.

[0181] Figure 4 A structural block diagram of a computing device 400 according to an embodiment of this application is shown. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0182] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0183] In one embodiment of this application, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0184] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 400 can also be a mobile or stationary server.

[0185] The processor 420 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the page operation method based on the large language model described above.

[0186] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the page operation method based on the large language model described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the page operation method based on the large language model described above.

[0187] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the page operation method based on the large language model described above.

[0188] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the page operation method based on the large language model described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the page operation method based on the large language model described above.

[0189] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the page operation method based on the large language model described above.

[0190] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the page operation method based on the large language model described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the page operation method based on the large language model described above.

[0191] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0192] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0193] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0194] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0195] The preferred embodiments disclosed above are merely illustrative of this application. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this application. These embodiments are selected and specifically described in this application to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A page operation method based on a large language model, characterized in that, include: Receive page operation requirement information for the target business system, wherein the page operation requirement information includes operation function information, which is the utility function that the target business system needs to use during the operation process; Obtain the current business page of the target business system, obtain historical operation information and model expectation information for the target business system, wherein the model expectation information determines the return information of the model; concatenate the page operation requirement information, the current business page, the historical operation information and the model expectation information to generate page operation text; Input the page operation text into the page parsing big model, and obtain the current page description information and current operation information output by the page parsing big model; Based on the current page description information and the page operation requirement information, it is determined whether the page operation termination condition is met, wherein the page operation termination condition is the condition for whether the page automated operation task has ended, as determined by the page operation requirement information. If the current page description information does not meet the page operation termination condition, the current business page is processed according to the current operation information, and current operation description information is generated according to the current operation information; the historical operation information is updated according to the current operation description information, wherein the historical operation information records operation description information for the target business system; If the current page description information meets the page operation termination conditions, the processing for the target business system shall be stopped.

2. The method as described in claim 1, characterized in that, Receiving page operation request information for the target business system also includes: Receive business purpose information and business parameter information for the target business system.

3. The method as described in claim 1, characterized in that, Obtaining the current business page of the target business system includes: Use a screenshot tool to capture the current business page of the target business system.

4. The method as described in claim 1, characterized in that, Determine whether the page operation termination condition is met based on the current page description information and the page operation requirement information, including: Obtain the business purpose information from the page operation requirement information; Determine whether the current page description information meets the business objective information; If so, then the page operation termination condition is met; If not, then the page operation termination condition has not been met.

5. The method as described in claim 1, characterized in that, Processing the current business page based on the current operation information includes: Generate current page operation instructions based on the current operation information; The current business page is processed based on the current page operation instructions, so that the target business system generates a new current business page.

6. The method as described in claim 1, characterized in that, The method further includes: Establish a communication connection between the target business system and the large page parsing model.

7. A page operation device based on a large language model, characterized in that, include: The receiving module is configured to receive page operation requirement information for the target business system, wherein the page operation requirement information includes operation function information, which is the utility function that the target business system needs to use during the operation process; The generation module is configured to obtain the current business page of the target business system, obtain historical operation information and model expectation information for the target business system, wherein the model expectation information determines the return information of the model; and concatenate the page operation requirement information, the current business page, the historical operation information and the model expectation information to generate page operation text. The acquisition module is configured to input the page operation text into the page parsing big model, and acquire the current page description information and current operation information output by the page parsing big model; The judgment module is configured to determine whether the page operation termination condition is met based on the current page description information and the page operation requirement information, wherein the page operation termination condition is the condition for whether the page automated operation task has ended as determined by the page operation requirement information. The operation module is configured to, when the current page description information does not meet the page operation termination condition, process the current business page according to the current operation information and generate current operation description information according to the current operation information; update the historical operation information according to the current operation description information, wherein the historical operation information records operation description information for the target business system; and stop processing for the target business system when the current page description information meets the page operation termination condition.

8. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 6.

Images