Visual big language model-based webpage navigation method and device, medium and product

Abstract

The embodiment of the invention provides a webpage navigation method and device based on a visual large language model, a medium and a product. The method comprises the following steps: receiving total task information needing to be navigated by a user, and determining first information associated with the total task information; inputting the first information into a first visual large language model for sub-task prediction to obtain a next sub-task which is output by the first visual large language model and needs to be executed; obtaining second information associated with the next subtask, inputting the second information into the second visual large language model predicted in the step, and obtaining the current execution action and the next execution step information corresponding to the subtask output by the second visual large language model; and under the condition that the next execution step information indicates that the execution of the sub-tasks is completed, returning to the step 2 until all the sub-tasks output by the second visual large language model are completed. According to the scheme, the problem of poor image recognition and positioning accuracy of the graphical user interface can be solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a webpage navigation method, device, medium and product based on a visual large language model. Background Art

[0002] Graphical User Interface (GUI) navigation algorithms require advanced screen interpretation, action reasoning, and positioning capabilities. Vision-Language Models (VLMs) can integrate multi-sensory skills to achieve general intelligence, so VLMs can be used to complete GUI navigation tasks. However, the existing application methods use visual methods such as optical character recognition (OCR) to obtain text, icons, and other data in GUI screenshots as visual information supplements for VLM models. However, they cannot be associated with specific tasks and step content, resulting in a lack of model evolution ability; they ignore the modular properties of GUI navigation tasks, resulting in low accuracy of reasoning results; and when the resolution of GUI images is too large, there are errors in the recognition of targets in the image. Summary of the invention

[0003] At least one embodiment of the present application provides a webpage navigation method, device, medium and product based on a visual large language model, which is used to solve the problem of errors in target recognition of images when the VLM model used in the prior art has poor capabilities or the GUI image resolution is too large.

[0004] In order to solve the above technical problems, this application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a webpage navigation method based on a visual large language model, comprising:

[0006] Step a, receiving the general task information that the user needs to navigate, and determining the first information associated with the general task information;

[0007] Step b, inputting the first information into a first visual large language model for subtask prediction, and obtaining a next subtask to be executed output by the first visual large language model;

[0008] Step c, obtaining second information associated with the next subtask, inputting the second information into the second visual large language model of step prediction, and obtaining the current execution action and next execution step information corresponding to the subtask output by the second visual large language model;

[0009] Step d. When the next execution step information indicates that the subtask is completed, update the first information to the third information, and return to step b until the next execution step information of the target subtask output by the second visual large language model indicates that the target subtask is completed; the target subtask is the last subtask that needs to be executed output by the first visual large language model.

[0010] Optionally, the step a comprises:

[0011] According to the total task information and the first case database, obtaining a total task reference case, a historical subtask sequence and a current subtask-step sequence associated with the total task information;

[0012] Get the current first graphical user interface GUI screenshot;

[0013] The total task information, the total task reference case, the historical subtask sequence, the current subtask-step sequence and the first GUI screenshot are determined as the first information.

[0014] Optionally, the step c comprises:

[0015] According to the subtask and the second case database, obtaining a subtask reference case and a sequence of historical steps-current steps associated with the subtask;

[0016] Obtain a second GUI screenshot of the current step of the subtask;

[0017] Using a visual target detection algorithm, obtaining information about a region of interest in the second GUI screenshot;

[0018] Determine a maximum contour interception area according to the interest area information and the second GUI screenshot;

[0019] Determine the subtask reference case, the subtask, the sequence of the historical step-current step, the maximum contour interception area and the interest area information as the second information;

[0020] Generate a first input instruction according to the second information and a preset first instruction generator;

[0021] The first input instruction is input into the second visual large language model to obtain the current execution action and next execution step information corresponding to the subtask output by the second visual large language model.

[0022] Optionally, using a visual target detection algorithm to obtain information about a region of interest associated with the second GUI screenshot includes:

[0023] Using the second case database, obtaining a target-related case associated with the current step;

[0024] Determine case screenshots and case execution actions of the target-related cases;

[0025] Input the case screenshot and the case execution action into a preset model having the visual target detection algorithm, and obtain a target detection box related to the target-related case in the second GUI screenshot;

[0026] The target detection frame is used to extract the region of interest of the second GUI screenshot to obtain the region of interest information of the second GUI screenshot.

[0027] Optionally, after step c, the method further comprises:

[0028] If the next execution step information is used to indicate that the subtask is completed, then executing the step of using the second visual large language model to re-receive the subtask output by the first visual large language model;

[0029] If the next execution step information is used to indicate that the subtask has failed, the predicted failed subtask-step sequence is used as input information, and the input information is input into the first visual large language model to regenerate the current subtask.

[0030] Optionally, after step c, the method further comprises:

[0031] If the next execution step information is used to represent other information other than the completion of the subtask and the failure of the subtask, the predicted failed subtask-step sequence is executed as input information, and the input information is input into the first visual large language model to regenerate the current subtask.

[0032] Optionally, before step a, the method further comprises:

[0033] Utilize multiple historical case data sets to build a case database;

[0034] Each historical case in the historical case data set includes a plurality of overall task description information and a subtask set associated with each overall task description information;

[0035] The subtask set includes subtask description information and feature information associated with the subtask description information;

[0036] The feature information includes the execution step description information of the subtask decomposition, action content, and GUI screenshots; the action content includes the action type, location information, and action value.

[0037] In a second aspect, an embodiment of the present application provides a webpage navigation device based on a visual large language model, comprising:

[0038] A first processing module is used to receive the general task information that the user needs to navigate, and determine the first information associated with the general task information;

[0039] A second processing module is used to input the first information into a first visual large language model predicted by a subtask, and obtain a next subtask to be executed output by the first visual large language model;

[0040] A third processing module is used to obtain second information associated with the next subtask, input the second information into a second visual large language model for step prediction, and obtain the current execution action and next execution step information corresponding to the subtask output by the second visual large language model;

[0041] The fourth processing module is used to update the first information to the third information when the next execution step information indicates that the subtask is completed, and return to step b until the next execution step information of the target subtask output by the second visual large language model indicates that the target subtask is completed; the target subtask is the last subtask that needs to be executed output by the first visual large language model.

[0042] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0043] In a fourth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0044] Compared with the prior art, the webpage navigation method, device, medium and product based on the visual large language model provided by the embodiment of the present application include steps a to d, step a is used to receive the total task information that the user needs to navigate, and determine the first information associated with the total task information; step b inputs the first information into the first visual large language model predicted by the subtask, and obtains the next subtask to be executed output by the first visual large language model; step c obtains the second information associated with the next subtask, inputs the second information into the second visual large language model predicted by the step, and obtains the current execution action and the next execution step information corresponding to the subtask output by the second visual large language model; after each step c is executed, step d determines that the process ends when the next execution step information of the last subtask is executed, otherwise steps b and c are re-executed. The present application realizes the overall process of navigation through subtask prediction and execution step prediction, which not only ensures the invariance of the model structure, but also solves the problems of too large resolution of GUI images, poor accuracy of recognition and positioning of tiny text and icons, etc.; the task decomposition control flow is used to effectively improve the accuracy of step reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0046] Figure 1 A schematic diagram of a flow chart of a webpage navigation method based on a visual large language model provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a process for predicting subtasks provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a process flow for predicting the steps provided in the embodiments of the present application;

[0049] Figure 4 A schematic diagram of the process of visual target detection provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of the overall process of a webpage navigation method based on a visual large language model provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of the structure of a webpage navigation device based on a visual large language model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0053] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment. "And / or" in the specification and claims represents at least one of the connected objects.

[0054] In order to enable those skilled in the art to better understand the embodiments of the present application, the present application first describes the shortcomings of the three methods of the related art as follows:

[0055] Disadvantages of related technologies 1: When using visual algorithms such as OCR to extract text and icon data as information supplements for VLM, related technologies have the following problems: they cannot be associated with specific tasks and step contents, and the acquired text and icon data have redundancy and interference problems; they cannot be associated with relevant case steps, and the model lacks the ability to evolve autonomously. This application designs a case database and an interactive visual prompt model, and uses the RAG method to retrieve GUI images and action information of relevant tasks and steps from the case database as visual prompts. The interactive visual prompt model is used to detect similar instances in the current GUI image and provide them to VLM as information supplements, which not only simplifies the information content, but also improves the algorithm's ability to learn and evolve autonomously.

[0056] The second shortcoming of the related technology: When the related technology uses VLM to decompose the steps of GUI navigation tasks, it adopts the CoT and ReAct methods, without considering the modular properties of GUI navigation tasks. For example, the task of "ordering a cloud host ECS on the mobile cloud website, configured as..." can be decomposed into four subtasks: "configuration analysis", "cloud host ECS search", "cloud host ECS configuration", and "cloud host ECS purchase". Each subtask can be decomposed into a series of operation steps, and these subtasks have modular action cases, which can be used as a reference to ensure the accuracy of step decomposition. This application fully considers the modular properties of GUI navigation tasks and designs the control flow of task decomposition, including two stages: subtask prediction and step prediction.

[0057] Disadvantage three of related technologies: In dealing with the problem of excessively large GUI image resolution, related technologies either directly reduce the resolution to the input size of the VLM visual branch (mostly 224×224, 384×384), or modify the VLM model structure to accept larger resolution image inputs, such as CogAgent; the first method will lose tiny icons, buttons, and text information, resulting in a decrease in the accuracy of recognition and positioning, and the second method is too dependent on the structure of VLM and cannot be migrated to other pre-trained VLM models. This application uses an interactive visual cue model to detect the location of relevant target instances, performs maximum contour interception and ROI extraction on the original GUI image, and uses this as the input of the VLM visual branch, thereby improving the recognition and positioning accuracy of tiny targets without changing the model structure.

[0058] The embodiments of the present application provide a webpage navigation method, device, medium and product based on a visual large language model, which can solve the shortcomings or problems existing in the background technology or the above-mentioned related technologies. Among them, the method and the device are based on the same application concept. Since the principles of solving the problem by the method and the device are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated.

[0059] Please refer to Figure 1 , a webpage navigation method based on a visual large language model provided in an embodiment of the present application includes:

[0060] Step a, receiving the general task information that the user needs to navigate, and determining the first information associated with the general task information;

[0061] Step b, inputting the first information into a first visual large language model for subtask prediction, and obtaining a next subtask to be executed output by the first visual large language model;

[0062] Step c, obtaining second information associated with the next subtask, inputting the second information into the second visual large language model of step prediction, and obtaining the current execution action and next execution step information corresponding to the subtask output by the second visual large language model;

[0063] Step d. When the next execution step information indicates that the subtask is completed, update the first information to the third information, and return to step b until the next execution step information of the target subtask output by the second visual large language model indicates that the target subtask is completed; the target subtask is the last subtask that needs to be executed output by the first visual large language model.

[0064] It should be noted that when the first information is updated to the third information, not all information is updated. For example, the first information includes the current GUI screenshot, historical sequence information, and total task information. When the first information is updated, only the current GUI screenshot and historical sequence information are updated. Here, the historical sequence information includes the historical sequence information corresponding to the previous subtask.

[0065] In an embodiment of the present application, the total task information that the user needs to navigate is processed and analyzed to obtain input information for determining the input of the first visual large language model. After the first information is input, the first visual large language model is used to predict the steps of the next subtask, and the second information corresponding to the next subtask to be executed is input into the second visual large language model. When the second visual large language model performs step prediction, the meaning of the step output each time is different, until the next execution step information indicates that the next execution step information indicates that the subtask has been completed, indicating that the current subtask has completed all steps, and the first information is updated according to the next execution step information, updated to the third information, and then steps b and c are re-executed until step d outputs the next execution step information of the target subtask to indicate that the target subtask has been executed. The target subtask is the last subtask that needs to be executed output by the first visual large language model.

[0066] For example, if the total task input by the user is the total task of "buying a piece of clothing", this application will decompose "buying a piece of clothing" into three subtasks of "checking in to shopping software A", "searching for clothing", and "buying". Each subtask may correspond to multiple steps. For example, "searching for clothing" at least includes "the step of finding the search box", "the step of inputting into the search box", and "the step of selecting". This application will complete the execution of the entire total task when the next execution step information of "buying" indicates that the subtask has been completed.

[0067] This application decomposes the above-mentioned general task into subtasks, and then decomposes the subtasks into steps. Through the subtask-step decomposition control flow, the accuracy of step reasoning can be effectively improved.

[0068] It should be noted that the visual large language model VLM involved in the above-mentioned execution prediction module and subtask prediction module is a unified multimodal large model that has been fine-tuned for multiple tasks. It is not limited to a specific model structure and supports most open source baseline models, such as GPT-4V, LVLMs, CogVLM, Qwen-VL, etc.

[0069] Optionally, before step a, the method further comprises:

[0070] Utilize multiple historical case data sets to build a case database;

[0071] Each historical case in the historical case data set includes a plurality of overall task description information and a subtask set associated with each overall task description information;

[0072] The subtask set includes subtask description information and feature information associated with the subtask description information;

[0073] The feature information includes the execution step description information of the subtask decomposition, action content, and GUI screenshots; the action content includes the action type, location information, and action value.

[0074] In an embodiment of the present application, before step a, multiple historical case data sets are obtained to construct a corresponding case database. For example, the historical case data set of the overall task-subtask can be used to construct a first case database about the subtask; or the second case database about the step can be constructed based on the historical case data set of the subtask-step.

[0075] Specifically, the case database can be represented by the following information:

[0076] C=Set(C1,C2,...,C n );

[0077] C i ={T i ,{ST1,ST2,...,ST m}},i=1,2,...,n;

[0078]

[0079] Among them, C represents the case data set, C i Indicates each actual case, T i Indicates the overall task description text of the case, ST iIndicates the information of the ith subtask in this case, x i Represents the description text of the i-th subtask, Indicates the step description text corresponding to the j-th execution step of the i-th subtask decomposition, Indicates the corresponding action content, including type Location Information and action value Indicates the corresponding GUI screenshot.

[0080] This application utilizes the construction of a case database, which is beneficial to supplement the information of the subsequent visual language large model, thereby enhancing the action positioning capability of GUI navigation, and effectively associating the case library can enhance the algorithm's autonomous learning and evolution capabilities.

[0081] Optionally, the above step a includes:

[0082] According to the total task information and the first case database, obtaining a total task reference case, a historical subtask sequence and a current subtask-step sequence associated with the total task information;

[0083] Get the current first graphical user interface GUI screenshot;

[0084] The total task information, the total task reference case, the historical subtask sequence, the current subtask-step sequence and the first GUI screenshot are determined as the first information.

[0085] In an embodiment of the present application, the above-mentioned method is used to construct a first case database, and the first case database is a case database associated with the total task-subtask. According to the total task information and the first case database, the total task reference case, the historical subtask sequence and the current subtask-step sequence associated with the total task information are obtained, and the current first GUI screenshot is obtained. The first GUI screenshot can be the original GUI screenshot without considering the detailed information. The present application determines the total task information, the total task reference case, the historical subtask sequence, the current subtask-step sequence and the first GUI screenshot as the first information, and the first information is used as the input information of the subsequent first visual large language model.

[0086] In a specific implementation, referring to Figure 2 The flowchart of the subtask prediction algorithm is shown in Figure 1. Assume that the total task is T and the historical subtask sequence is ST1, ST2, ..., ST t-1 , the current subtask is ST t , the corresponding sequence of steps is S1,S2,...,S t , the current GUI screenshot is I t , the next subtask is STt+1 The algorithm flow is as follows: First, the most similar K related reference cases C are retrieved from the first case database with the help of retrieval augmentation generation (RAG). q , C q =(C1,C2,...,C K )=RAG(C,T,ST t );Use the first instruction generator Instr_Gen2 to construct the first input instruction Instr of the first visual language model 2,t : The first input instruction is expressed as:

[0087] Instr 2,t =Instr_Gen2(C q ,T,(ST1,ST2,...,ST t-1 ),(ST t ,(S1,S2,...,S t )),I t );

[0088] Finally, the first visual language model is used to generate the next subtask: ST t+1 =VLM(Instr 2,t ). Here, the visual input of this stage is the original GUI screenshot, and there is no need to consider the detailed information, because the output of the large model at this stage is the prediction of the subtask and does not involve the problem of action localization.

[0089] Optionally, the step c comprises:

[0090] According to the subtask and the second case database, obtaining a subtask reference case and a sequence of historical steps-current steps associated with the subtask;

[0091] Obtain a second GUI screenshot of the current step of the subtask;

[0092] Using a visual target detection algorithm, obtaining information about a region of interest in the second GUI screenshot;

[0093] Determine a maximum contour interception area according to the interest area information and the second GUI screenshot;

[0094] Determine the subtask reference case, the subtask, the sequence of the historical step-current step, the maximum contour interception area and the interest area information as the second information;

[0095] Generate a first input instruction according to the second information and a preset first instruction generator;

[0096] The first input instruction is input into the second visual large language model to obtain the current execution action and next execution step information corresponding to the subtask output by the second visual large language model.

[0097] In an embodiment of the present application, the second case database is a case database of subtasks-steps determined according to the above method. The present application utilizes the second case database to obtain subtask reference cases and a sequence of historical steps-current steps associated with the current subtask. The present application obtains a second GUI screenshot, and utilizes a visual target detection algorithm to obtain region of interest information related to the second GUI screenshot; based on the region of interest information and an image segmentor, the second GUI screenshot is subjected to image segmentation to determine the maximum contour interception region corresponding to the second GUI screenshot. The subtask reference case, the subtask, the sequence of historical steps-current steps, the maximum contour interception region, and the region of interest information are determined as the second information, and a first instruction generator is used to generate a first input instruction for input into the second visual large language model. The first input instruction is input into the second visual large language model to obtain the current execution action and next execution step information corresponding to the subtask output by the second visual large language model.

[0098] This application uses GUI interest regions, target detection boxes and maximum contour interception areas to replace original GUI screenshots as the visual input of VLM, which not only ensures the invariance of the model structure, but also solves problems such as excessive GUI image resolution and poor accuracy in identifying and positioning tiny text and icons.

[0099] In a specific implementation, referring to Figure 3 In the step prediction method flow chart shown in FIG. 1 , it is assumed that the subtask is ST and the corresponding current step is S t , the historical steps are S1,S2,...,S t-1 , the second GUI screenshot is the current GUI screenshot t Specifically, the most similar K related reference cases C are retrieved from the second case database using the RAG method. q :C q =(C1,C2,...,C K )=RAG(C,ST,S t ); Interactive visual cue model using visual object detection algorithm Model IVP (Interactive Visual Prompt Model), get the GUI screenshot area of ​​interest information associated with the subtask and the current step (Patches t ) and target detection boxes t ;

[0100] It is expressed by the following formula: t ,Patches t )=Moudle vod (C,ST,S t ,I t );

[0101] With the help of the image segmentation tool Image_Crop, t Perform maximum contour interception to obtain the maximum contour interception area I_max_crop t , the maximum contour interception area is expressed as:

[0102] I_max_crop t =Image_Crop(I t ,Boxes t );

[0103] Using the first instruction generator Instr_Gen1, the first input instruction Instr of the second visual language model 1,t .

[0104] The first input instruction Instr 1,t =

[0105] Instr Gen1 (C q ,ST,S t ,(S1,S2,…,S t-1 ),I_max_crop t ,Boxes t ,Patches t ). The first input instruction Instr t Input into the unified fine-tuned second visual language model to infer the current execution action A t and the next step S t+1 :

[0106] (A t ,S t+1 )=VLM(Instr 1,t );

[0107] In the process of executing step prediction, the visual target detection algorithm is used to obtain the GUI screenshot interest area data associated with the subtask and the current step, and provided to the second visual large language model as information supplement. Compared with the use of visual algorithms such as OCR to extract the text and icon data of the GUI screenshot as information supplement, it not only simplifies the information content, but also associates the case library, and improves the algorithm's ability to learn and evolve autonomously. At the same time, this application uses interest area Patches, target detection boxes, and maximum contour interception areas as visual inputs instead of the original GUI screenshots, which is more conducive to the processing of detailed features and solves the problem of too large GUI image resolution and poor accuracy in identifying and positioning tiny text and icons.

[0108] Optionally, using a visual target detection algorithm to obtain information about a region of interest associated with the second GUI screenshot includes:

[0109] Using the second case database, obtaining a target-related case associated with the current step;

[0110] Determine case screenshots and case execution actions of the target-related cases;

[0111] Input the case screenshot and the case execution action into a preset model having the visual target detection algorithm, and obtain a target detection box related to the target-related case in the second GUI screenshot;

[0112] The target detection frame is used to extract the region of interest of the second GUI screenshot to obtain the region of interest information of the second GUI screenshot.

[0113] In an embodiment of the present application, the RAG method is used to obtain the target-related case associated with the subtask-current step from the second case database, and the target-related case includes the corresponding case screenshot and case execution action. The case screenshot and the case execution action are input into a preset model that has been trained and has the visual target detection algorithm, and the target detection box related to the target-related case in the second GUI screenshot is obtained. The target detection box is used to extract the region of interest ROI ((Region Of Interest)) to generate region of interest Patches.

[0114] Specifically, refer to Figure 4 The process of visual target detection shown in FIG. 1 includes: assuming that the subtask to be processed is ST, the corresponding current step is S, and the current GUI screenshot is I. Step 1 Retrieve the GUI screenshots of the relevant tasks and steps from the case database C using the RAG method and action information

[0115]

[0116] Taking screenshots with GUI and action information The result is used as a visual cue, using the trained interactive visual cue model Model IVP (Interactive Visual Prompt Model), get the Top K similar target detection boxes in the current GUI screenshot:

[0117]

[0118] With the help of the target detection box, ROI (Region Of Interest) is extracted from the GUI screenshot to generate the region of interest Patches: Patches = (patch1, patch2, ..., patch K )=ROI(Boxes,I).

[0119] In an optional embodiment, after step c, the method further comprises:

[0120] If the next execution step information is used to indicate that the subtask is completed, then executing the step of using the second visual large language model to re-receive the subtask output by the first visual large language model;

[0121] If the next execution step information is used to indicate that the subtask has failed, the predicted failed subtask-step sequence is used as input information, and the input information is input into the first visual large language model to regenerate the current subtask.

[0122] In an optional embodiment, after step c, the method further comprises:

[0123] If the next execution step information is used to represent other information other than the completion of the subtask and the failure of the subtask, the predicted failed subtask-step sequence is executed as input information, and the input information is input into the first visual large language model to regenerate the current subtask.

[0124] In an embodiment of the present application, the direction of the entire processing flow is determined according to the content of the next step generated in the step prediction stage, thereby realizing the process of ending reasoning, correcting reasoning and continuing reasoning: when the next step is "completed", the step prediction stage is jumped out, and the subtask prediction stage is re-entered to generate the next subtask; when the next step is "failed", the step prediction stage is jumped out, and the subtask prediction stage is re-entered, and the subtask and step sequence predicted to fail are used as input information to regenerate the current subtask; when the next step indicates other information other than subtask completion and subtask failure, the step prediction process is executed normally.

[0125] In a specific embodiment, the present application proposes a GUI webpage navigation system (not shown) based on a visual language large model driven system, including a subtask prediction module, a step prediction module, a control logic unit, etc. Figure 5 In the overall flow diagram shown, when the webpage navigation system receives a user's general task request, the entire end-to-end processing flow includes two stages and a control logic unit:

[0126] Subtask prediction phase: This process takes the overall task description text, current GUI screenshots, historical subtask sequences and other information as input, and uses the fine-tuned first visual language large model to infer the next subtask;

[0127] Step prediction phase: This process takes subtask description text, current GUI screenshot, historical step sequence and other information as input, and uses the fine-tuned second visual language large model to infer the current execution action and the next step;

[0128] Control logic unit: Determines the direction of the entire processing flow according to the content of the next step generated in the step prediction stage, thereby realizing the process of reasoning end, reasoning error correction and reasoning continuation: when the next step is "completed", it jumps out of the step prediction stage, re-enters the subtask prediction stage, and generates the next subtask; when the next step is "failed", it jumps out of the step prediction stage, re-enters the subtask prediction stage, uses the subtask and step sequence that failed to be predicted as input information, and regenerates the current subtask; when the next step is other information, the step prediction process is executed normally.

[0129] It should be noted that the design of the task decomposition control flow fully considers the modular properties of the GUI navigation task. In the process of processing the subtask step prediction, similar action cases can be used as a reference to greatly improve the accuracy of the reasoning results.

[0130] In summary, the solution of the present application can make up for the shortcomings of the existing technical solutions, and is a low-cost, feasible and easy-to-promote GUI navigation technology. The solution of the present application obtains the interest region data associated with the GUI task by constructing a visual target detection module, and provides it to the VLM as information supplement, which simplifies the information content while strengthening the action positioning ability of the GUI navigation, and can effectively associate the case library to improve the algorithm's autonomous learning and evolution capabilities; the solution of the present application introduces the modular properties of the GUI navigation task and designs a task decomposition control flow algorithm. In the process of processing the step prediction of the subtask, similar modular action cases can be used as a reference to greatly improve the accuracy of the reasoning results; at the same time, the solution of the present application innovatively uses the GUI interest region, target detection box and maximum contour interception area to replace the original GUI screenshot as the visual input of the VLM, which not only ensures the invariance of the model structure, but also solves the problem of poor accuracy in the recognition and positioning of tiny text and icons caused by the excessive resolution of the GUI image.

[0131] The solution of this application designs a case database and an interactive visual prompt model, and constructs a visual target detection module to obtain the region of interest data associated with the GUI task and provide it to the VLM as information supplement. Compared with visual auxiliary methods such as OCR, this method simplifies the information content while strengthening the action positioning ability of GUI navigation, and can effectively associate the case library to improve the algorithm's autonomous learning and evolution capabilities.

[0132] The solution of this application introduces the modular properties of GUI navigation tasks and designs a task decomposition control flow algorithm, which mainly includes three components: subtask prediction, execution step prediction, and control logic unit. In the process of processing subtask step prediction, similar modular action cases can be used as reference to greatly improve the accuracy of reasoning results.

[0133] The solution of the present application utilizes GUI regions of interest, target detection boxes, and maximum contour interception areas instead of original GUI screenshots as the visual input of the VLM, which not only ensures the invariance of the model structure, but also solves the problems of excessive GUI image resolution and poor accuracy in identifying and positioning tiny text and icons.

[0134] The above describes various methods of the embodiments of the present application. The following further provides devices for implementing the above methods.

[0135] Please refer to Figure 6 The embodiment of the present application also provides a webpage navigation device based on a visual large language model, including:

[0136] The first processing module 61 is used to receive the general task information that the user needs to navigate, and determine the first information associated with the general task information;

[0137] A second processing module 62 is used to input the first information into a first visual large language model for subtask prediction, and obtain a next subtask to be executed output by the first visual large language model;

[0138] The third processing module 63 is used to obtain second information associated with the next subtask, input the second information into the second visual large language model for step prediction, and obtain the current execution action and next execution step information corresponding to the subtask output by the second visual large language model;

[0139] The fourth processing module 64 is used to update the first information to the third information when the next execution step information indicates that the subtask is completed, and return to step b until the next execution step information of the target subtask output by the second visual large language model indicates that the target subtask is completed; the target subtask is the last subtask that needs to be executed output by the first visual large language model.

[0140] Optionally, the first processing module 61 includes:

[0141] A first processing unit, configured to obtain, according to the total task information and a first case database, a total task reference case, a historical subtask sequence, and a current subtask-step sequence associated with the total task information;

[0142] A first acquisition unit, used to acquire a current first graphical user interface GUI screenshot;

[0143] The second processing unit is used to determine the total task information, the total task reference case, the historical subtask sequence, the current subtask-step sequence and the first GUI screenshot as the first information.

[0144] Optionally, the third processing module 63 includes:

[0145] A second acquisition unit, configured to acquire, according to the subtask and a second case database, a subtask reference case and a sequence of historical steps-current steps associated with the subtask;

[0146] A third acquisition unit, used to acquire a second GUI screenshot of the current step of the subtask;

[0147] A fourth acquisition unit, configured to obtain information about a region of interest associated with the second GUI screenshot using a visual target detection algorithm;

[0148] A third extraction unit is used to determine a maximum outline interception area according to the interest area information and the second GUI screenshot;

[0149] The fourth extraction unit is used to determine the subtask reference case, the subtask, the sequence of the historical step-current step, the maximum contour interception area and the interest area information as the second information;

[0150] a fifth processing unit, configured to generate a first input instruction according to the second information and a preset first instruction generator;

[0151] The sixth extraction unit is used to input the first input instruction into the second visual large language model to obtain the current execution action and next execution step information corresponding to the subtask output by the second visual large language model.

[0152] Optionally, the fourth acquiring unit is specifically configured to:

[0153] Using the second case database, obtaining a target-related case associated with the current step;

[0154] Determine case screenshots and case execution actions of the target-related cases;

[0155] Input the case screenshot and the case execution action into a preset model having the visual target detection algorithm, and obtain a target detection box related to the target-related case in the second GUI screenshot;

[0156] The target detection frame is used to extract the region of interest of the second GUI screenshot to obtain the region of interest information of the second GUI screenshot.

[0157] Optionally, the webpage navigation device based on the visual large language model provided in the embodiment of the present application further includes:

[0158] a fifth processing module, configured to execute, if the next execution step information is used to indicate that the subtask is completed, a step of re-receiving the subtask output by the first visual large language model using the second visual large language model;

[0159] The sixth processing module is used to execute the predicted failed subtask-step sequence as input information if the next execution step information is used to indicate that the subtask has failed, and input the input information into the first visual large language model to regenerate the current subtask.

[0160] Optionally, the webpage navigation device based on the visual large language model provided in the embodiment of the present application further includes:

[0161] The seventh processing module is used to execute the predicted failed subtask-step sequence as input information if the next execution step information is used to represent other information other than the completion of the subtask and the failure of the subtask, and input the input information into the first visual large language model to regenerate the current subtask.

[0162] Optionally, the webpage navigation device based on the visual large language model provided in the embodiment of the present application further includes:

[0163] A construction module, used to construct a case database using multiple historical case data sets;

[0164] Each historical case in the historical case data set includes a plurality of overall task description information and a subtask set associated with each overall task description information;

[0165] The subtask set includes subtask description information and feature information associated with the subtask description information;

[0166] The feature information includes the execution step description information of the subtask decomposition, action content, and GUI screenshots; the action content includes the action type, location information, and action value.

[0167] It should be noted that the device in this embodiment is a device corresponding to the webpage navigation method based on the visual large language model, and the implementation methods in the above embodiments are all applicable to the embodiments of the device, and can achieve the same technical effect. The above device provided in the embodiment of the present application can implement all the method steps implemented in the above method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0168] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the webpage navigation method embodiment based on the visual large language model is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0169] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0170] It should be noted that, in this article, the terms "include", "include" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0171] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0172] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A webpage navigation method based on a visual large language model, characterized in that: include: Step a, receiving the general task information that the user needs to navigate, and determining the first information associated with the general task information; Step b, inputting the first information into a first visual large language model for subtask prediction, and obtaining a next subtask to be executed output by the first visual large language model; Step c, obtaining second information associated with the next subtask, inputting the second information into the second visual large language model of step prediction, and obtaining the current execution action and next execution step information corresponding to the subtask output by the second visual large language model; Step d, when the next execution step information indicates that the subtask is completed, updating the first information to the third information, and returning to step b, until the second visual language model outputs the next execution step information of the target subtask indicating that the target subtask is completed; The target subtask is the last subtask output by the first large visual language model that needs to be executed.

2. The method according to claim 1, characterized in that The step a comprises: According to the total task information and the first case database, obtaining a total task reference case, a historical subtask sequence and a current subtask-step sequence associated with the total task information; Get the current first graphical user interface GUI screenshot; The total task information, the total task reference case, the historical subtask sequence, the current subtask-step sequence and the first GUI screenshot are determined as the first information.

3. The method according to claim 1, characterized in that The step c comprises: According to the subtask and the second case database, obtaining a subtask reference case and a sequence of historical steps-current steps associated with the subtask; Obtain a second GUI screenshot of the current step of the subtask; Using a visual target detection algorithm, obtaining information about a region of interest in the second GUI screenshot; Determine a maximum contour interception area according to the interest area information and the second GUI screenshot; Determine the subtask reference case, the subtask, the sequence of the historical step-current step, the maximum contour interception area and the interest area information as the second information; Generate a first input instruction according to the second information and a preset first instruction generator; The first input instruction is input into the second visual large language model to obtain the current execution action and next execution step information corresponding to the subtask output by the second visual large language model.

4. The method according to claim 3, characterized in that Using a visual target detection algorithm, obtain the region of interest information of the second GUI screenshot, including: Using the second case database, obtaining a target-related case associated with the current step; Determine case screenshots and case execution actions of the target-related cases; Input the case screenshot and the case execution action into a preset model having the visual target detection algorithm, and obtain a target detection box related to the target-related case in the second GUI screenshot; The target detection frame is used to extract the region of interest of the second GUI screenshot to obtain the region of interest information of the second GUI screenshot.

5. The method according to claim 1, characterized in that After step c, the method further comprises: If the next execution step information is used to indicate that the subtask is completed, then executing the step of using the second visual large language model to re-receive the subtask output by the first visual large language model; If the next execution step information is used to indicate that the subtask has failed, the predicted failed subtask-step sequence is used as input information, and the input information is input into the first visual large language model to regenerate the current subtask.

6. The method according to claim 5, characterized in that After step c, the method further comprises: If the next execution step information is used to represent other information other than the completion of the subtask and the failure of the subtask, the predicted failed subtask-step sequence is executed as input information, and the input information is input into the first visual large language model to regenerate the current subtask.

7. The method according to claim 1, characterized in that Before step a, the method further comprises: Utilize multiple historical case data sets to build a case database; Each historical case in the historical case data set includes a plurality of overall task description information and a subtask set associated with each overall task description information; The subtask set includes subtask description information and feature information associated with the subtask description information; The feature information includes the execution step description information of the subtask decomposition, action content, and GUI screenshots; the action content includes the action type, location information, and action value.

8. A webpage navigation device based on a visual large language model, characterized in that: include: A first processing module is used to receive the general task information that the user needs to navigate, and determine the first information associated with the general task information; A second processing module is used to input the first information into a first visual large language model predicted by a subtask, and obtain a next subtask to be executed output by the first visual large language model; A third processing module is used to obtain second information associated with the next subtask, input the second information into a second visual large language model for step prediction, and obtain the current execution action and next execution step information corresponding to the subtask output by the second visual large language model; a fourth processing module, configured to update the first information to third information when the next execution step information indicates that the subtask is completed, and return to step b until the second visual large language model outputs the next execution step information of the target subtask indicating that the target subtask is completed; The target subtask is the last subtask output by the first large visual language model that needs to be executed.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images