Webpage navigation system and method based on image marking

Abstract

The invention discloses a webpage navigation system and method based on image tagging, and the system comprises a perception model module which is used for predicting the next operation according to the user demand, historical operation and webpage information; the first checking module is used for judging whether the operation purpose of the current operation is completed or not and whether supplementary operation is needed or not by utilizing the MLLM according to the regional webpage screenshots before and after the current operation is executed; and the second checking module is used for checking whether the user demand is not completed or not according to the result output by the first checking module and the historical operation. By marking the webpage UI image, reducing the content of the DOM tree and improving the understanding of the MLLM on the webpage operation from different stages, the webpage navigation intelligent agent can predict the webpage operation more efficiently and more accurately.

Description

Technical Field

[0001] The present application relates to a webpage navigation system and method based on image marking, belonging to the field of software technology. Background Art

[0002] Web navigation services guide users to the mainstream business operations on the website and recommend potential web page elements that meet user needs. In recent years, with the development of large-scale language models LLM, models such as GPT3.5 and Qwen have strong text understanding capabilities. The web navigation agent developed based on LLM will help users plan and automatically perform web page operations based on the HTML or DOM tree of the web page. In more recent times, general multimodal large models MLLM, such as GPT4o, Claude3, Gork2, etc., have integrated images into the large model, further enhancing the ability of the agent to navigate the web page.

[0003] Professional web service MLLMs, such as Claude3.5 Computer Use and Ferret-UI, can directly return the location of web page elements according to operational requirements. However, the training methods of these models are special and they rarely provide service interfaces. Most companies cannot use these models to develop intelligent agents. General MLLMs cannot directly return the location of web page elements due to differences in training methods and can only roughly understand web page UI images. Most web page navigation intelligent agents are developed based on general MLLMs. Therefore, in order for the intelligent agent to select a specific web page element, the prompt word must provide a web page DOM tree.

[0004] Compared with web page HTML, web page DOM tree represents web page metadata in a more concise way. Most intelligent agents only use a small number of element features, such as type tags and text content, to represent web page elements in the DOM tree. Even with such a concise expression, the DOM tree still consumes a large amount of text prompt words; it is not only easy to cause common MLLM problems such as hallucinations and information loss, but also increases the time and cost of MLLM reasoning services, making web page navigation services slow. Summary of the invention

[0005] In view of at least one of the above technical problems, the present application provides a web page navigation system and method based on image tagging to improve the efficiency of web page navigation services.

[0006] Technical solution: To solve the above technical problems, the technical solution adopted in this application is:

[0007] In a first aspect, the present application provides a webpage navigation system based on image tags, the system comprising:

[0008] The perception model module is used to predict the next operation based on user needs, historical operations and web page information;

[0009] The first checking module is used to: use MLLM to determine whether the operation purpose of the current operation is completed and whether additional operations are required based on the screenshots of the regional webpage before and after the current operation is executed;

[0010] The second checking module is used to check whether the user demand is not completed according to the result output by the first checking module and the historical operation.

[0011] In some embodiments, the historical operations provide predicted and executed web page operations, each historical operation including a web page element DOM, an operation type, an operation purpose, and an execution status;

[0012] The web page information includes the current web page image and its web page element DOM;

[0013] The execution status includes a description of the required additional operations and / or unfinished content in the user requirements.

[0014] In some embodiments, the perception model module is specifically used to:

[0015] If the user's needs are not met;

[0016] Obtaining position information of the web page element in the previous operation step, and marking the operation element with a wireframe on the web page image in the previous operation step according to the position information to obtain a first image;

[0017] Fading the current web page image, and adding horizontal and vertical coordinate axes to the faded image to obtain a second image;

[0018] Based on the first image and the second image, the first multimodal large model MLLM is used to locate and obtain the coordinate information of the layout area where the next operation web page element is located, and expand it N times to obtain an extended coordinate area;

[0019] Obtaining DOM information of all interactive web page elements in the extended coordinate area; obtaining coordinate information of web page elements in the extended coordinate area, and drawing a position frame of each web page element on the web page image according to the coordinates; marking the position frame of the corresponding web page element according to the DOM information of each web page element, and obtaining a screenshot of the marked area web page;

[0020] According to the marked area web page screenshot, MLLM is used to determine the current operation web page element DOM, operation type and operation purpose from the DOM information of the web page element.

[0021] Furthermore, in some embodiments, the intervals between the horizontal and vertical coordinate axes are 200 pixels and 100 pixels respectively, and the two coordinate axes are located at the midpoints of the image in the horizontal and vertical directions respectively.

[0022] Furthermore, in some embodiments, the expanded coordinate area is obtained by expanding by 2 to 3 times.

[0023] In some embodiments, the first inspection module is specifically used to:

[0024] The web page image before the current operation is executed is cropped according to the extended coordinate area, and the web page elements to be executed are marked in the form of a wireframe to obtain a regional web page screenshot before the execution;

[0025] The web page image after the current operation is executed is cropped according to the extended coordinate area, and the executed web page elements are marked in the form of a wireframe to obtain a screenshot of the regional web page after the execution;

[0026] Based on the screenshots of the regional web pages before and after execution, the second multimodal large model MLLM is used to determine whether the operation purpose of the current operation is completed, whether additional operations are required, and if additional operations are required, the description of the additional operations is provided.

[0027] Furthermore, the step rule requirements of the second multimodal large model include: 1) if a drop-down window appears for the operation, it is determined that additional operations are required; 2) if the operation is a text input operation, whether the input content is correct is determined; 3) the image only shows the content of a partial area of ​​the web page, without considering the impact of the operation on the layout outside the screenshot.

[0028] In some embodiments, the second inspection module is specifically used to:

[0029] If additional operations are required, it is determined that the user's needs are not completed, and the description of the additional operations is added to the execution status of the historical operations, and the perception model module is returned to continue predicting the next operation;

[0030] If no additional operation is required, determine whether the user's needs are met based on historical operations;

[0031] If the user's needs are completed, the web page navigation process exits;

[0032] If the user's needs are not completed, the unfinished content will be added to the execution status of the historical operations, and the perception model module will be used to predict the next operation.

[0033] Furthermore, judging whether the user's needs are completed based on historical operations includes:

[0034] Summarize all web page images after each operation in the historical operation, mark the location of the executed web page elements with a wireframe on each web page image, and obtain the web page screenshots after all operations;

[0035] Based on historical operations and screenshots of web pages after all operations, determine whether user needs are completed, as well as the unfinished content if user needs are not completed.

[0036] In a second aspect, the present application provides a webpage navigation method based on image tags, the method comprising:

[0037] Obtain user needs, and execute loop steps until the user needs have been met; wherein the loop steps include:

[0038] S1. If the user's needs are not met, predict the next operation based on the user's needs, historical operations and web page information;

[0039] S2. Based on the screenshots of the regional webpage before and after the current operation, use MLLM to determine whether the operation purpose of the current operation has been completed and whether additional operations are required;

[0040] S3. Check whether the user's needs are not completed based on whether additional operations are required and historical operations;

[0041] If additional operations are required, it is determined that the user's needs are not completed, and the description of the additional operations required is added to the execution status of the historical operations, and the process returns to S1 to continue predicting the next operation;

[0042] If no additional operation is required, determine whether the user's needs are completed based on historical operations; if the user's needs are completed, the web page navigation process exits; if the user's needs are not completed, the unfinished content is added to the execution status of historical operations, and return to S1 to continue predicting the next operation.

[0043] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the second aspect.

[0044] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the second aspect when executing the computer program.

[0045] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the steps of the method described in the second aspect when executed by a processor.

[0046] Beneficial effects: The web page navigation system and method based on image tagging provided in the present application have the following advantages: by tagging web page UI images, the content of the DOM tree is reduced, and at the same time, the MLLM's understanding of web page operations is improved from different stages, allowing the intelligent agent to predict web page operations more efficiently and accurately, thereby improving the efficiency of web page navigation services. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a schematic diagram of an application of a webpage navigation method based on image marking according to an embodiment of the present application;

[0048] Figure 2 It is a schematic diagram of the process of a webpage navigation system based on image marking according to an embodiment of the present application;

[0049] Figure 3 A schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] The present application is further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and cannot be used to limit the protection scope of the present application.

[0051] Example 1: Figure 2 As shown, a webpage navigation system based on image tags, the system comprises:

[0052] A. Perception model module, used to predict the next operation based on user needs, historical operations and web page information.

[0053] It should be noted that the historical operations provide predicted and executed web page operations, each historical operation includes the web page element DOM, operation type, operation purpose and execution status; the web page information includes the current web page image and its web page element DOM; the execution status contains descriptions of operations that need to be supplemented and / or unfinished content in user needs.

[0054] The perception model module is responsible for predicting the next operation. The perception model module of this solution completes the prediction task in steps through system prompts and two user prompts. The two user prompts correspond to different web page images and output requirements. The perception model will output the results of the two steps according to the user prompts.

[0055] The system prompts consist of system tasks, user requirements, historical operations, system rules, and inference rules. System tasks require MLLM to complete the prediction task based on the information and requirements of each step. Historical operations provide predicted and executed web page operations. Each historical operation consists of web page element DOM, operation type, operation purpose, and execution status. System rules require MLLM to output results in a specified format according to the requirements of each step. Inference rules require that the operations predicted by MLLM cannot be repeated with historical operations, and if the execution status of the last historical operation requires additional operations, the previous operation is supplemented.

[0056] The first user prompt, the first step of the perception model, requires the MLLM to locate the layout position of the next web page element. In addition to the text prompt, the input to the MLLM also includes two web page images.

[0057] The web page image of the browser window is obtained through third-party libraries such as Playwright and Selenium. The first input image represents the web page operation in the previous step. The position information of the web page elements in the previous step is obtained through the interface of the third-party library. The operation elements are marked with wireframes on the web page image in the previous step according to the position information.

[0058] The second image represents the current web page and coordinates. Use OpenCV and other image libraries to fade the current web page image, and add coordinate axes to the faded image. The interval of the horizontal coordinate axis is 200 pixels and the vertical one is 100 pixels. The two coordinate axes are located at the midpoints of the image in the horizontal and vertical directions. In addition to the midpoint position, the coordinate axis can also be set at the upper left or lower right position of the image. Multiple web page images with coordinate axes can be drawn according to different positions, and all these images can be sent to MLLM, but in order to reduce the response time of MLLM, only one coordinate axis image is drawn here. The coordinate axis roughly marks the relative positions of different layouts on the web page image, which can assist MLLM in inferring the position information of web page elements.

[0059] Both images are scaled to around 960*960 pixels and then sent to the MLLM service.

[0060] The text prompt consists of step tasks, picture descriptions, step rules, and output formats. The step tasks require MLLM to locate the area of ​​the next operation web page element. The step rules require MLLM to return the approximate location information of the operation element based on the coordinate axis and describe the reason for selecting this element. The output format requires MLLM to directly return the area coordinate information, such as [100, 50, 200, 150], which are the minimum horizontal coordinate, the minimum vertical coordinate, the maximum horizontal coordinate, and the maximum vertical coordinate, respectively.

[0061] Based on the image and text prompts, MLLM will return the coordinate information of the layout area where the operation element is located. The coordinate information returned by the general MLLM will have a certain error. This method will enlarge the coordinate area by 2 to 3 times to ensure that the operation element is contained in the area.

[0062] According to the expanded coordinate area, the DOM information of all interactive web page elements in this area is obtained. By default, common operable element types such as input, span, strong, select, etc. are interactive elements, and some li or div types with text content are also interactive elements.

[0063] Get the coordinate information of the web page elements in the area, and draw the location box of each web page element on the web page image according to the coordinates. According to the DOM information of each web page element, such as "id:1, tag:input, text:xxx", get the element id and mark it in the upper left corner of the location box; get the type tag and mark it in the lower right corner of the location box. The position of the marked information cannot overlap with the location box of other elements or be too close ( Figure 3 ). This marking method allows MLLM to understand the correspondence between each element in the DOM information and each element on the image, allowing MLLM to understand the web page layout more intuitively. The marked web page screenshot is then sent to MLLM.

[0064] The text prompt consists of step tasks, regional image descriptions, regional DOM, operation types, step rules, and output formats. The step tasks require MLLM to select specific operation elements and operation types based on existing information. Regional DOM is the DOM information of all web page elements in the region. The DOM information of each web page element corresponds to the marked content on the image. The operation type describes the range of optional operations, such as mouse click, text input, text acquisition, hovering, etc., and prompts the element type suitable for each operation. The step rules require the selection of specific web page elements from the regional DOM, the selection of appropriate operation types from the operation type, and the description of the purpose of the operation. The output format requires the output of the operation type, DOM information, and operation purpose of the operation element in the form of a dictionary.

[0065] Therefore, in some embodiments, the perception model module is specifically used to: obtain the position information of the web page element in the previous operation, and mark the operation element with a wireframe on the web page image in the previous operation according to the position information to obtain a first image;

[0066] The current web page image is faded, and horizontal and vertical coordinate axes are added to the faded image to obtain a second image; wherein the intervals between the horizontal and vertical coordinate axes are 200 pixels and 100 pixels respectively, and the two coordinate axes are located at the midpoints of the image in the horizontal and vertical directions respectively;

[0067] According to the first image and the second image, the coordinate information of the layout area where the web page element to be operated in the next step is obtained by using MLLM positioning, and the area is expanded by 2 to 3 times to obtain an expanded coordinate area.

[0068] Get the DOM information of all interactive web page elements in the extended coordinate area;

[0069] Obtaining coordinate information of web page elements in the extended coordinate area, and drawing a position frame of each web page element on the web page image according to the coordinates; marking the position frame of the corresponding web page element according to the DOM information of each web page element, and obtaining a marked area web page screenshot;

[0070] According to the marked area web page screenshot, MLLM is used to determine the current operation web page element DOM, operation type and operation purpose from the DOM information of the web page element.

[0071] Based on the input image and prompt words, MLLM can accurately feedback the web page elements that need to be operated. Using regional DOM instead of global DOM greatly reduces the length of text prompt words, reduces performance and cost expenses; less DOM information improves the reasoning accuracy of MLLM. At the same time, because the web page elements in the region are limited, some more important element features can be retained, such as retaining element features such as class name class and label label in the element DOM, helping MLLM better understand the function of each element.

[0072] After the predicted operation is executed, the web page element DOM, operation type and operation purpose of the current operation will be saved as the web page element DOM, operation type and operation purpose of the previous operation in the history operation. After the operation, the web page will enter a new state, such as jumping to a new page, generating a pop-up window, changing the input value, etc. At this time, the first check module will check whether the current operation purpose has been achieved, such as "whether the value in the input box has changed", "whether further selection is required based on the drop-down window", etc. "Check operation supplement" is an important module for the successful operation of the web page navigation agent, which helps the agent detect the correctness of the prediction results of the perception model.

[0073] B. The first checking module is used to: use MLLM to determine whether the operation purpose of the current operation is completed and whether additional operations are required based on the screenshots of the regional webpage before and after the current operation is executed;

[0074] The first inspection module is also composed of system prompts and two user prompts. The system prompts are composed of system tasks, operation purposes and system rules. The system task requires MLLM to determine whether the operation purpose is achieved based on the information of each step. The system rules require the output of the results of each step according to the step requirements.

[0075] The user prompt for the first step consists of the step task, screenshot description, and output format. The step task is to analyze the state of the regional webpage screenshot before the current operation is performed. The size and extended coordinate area of ​​the regional webpage screenshot are used to crop the webpage image before the operation is performed according to the regional position, and mark the webpage elements to be executed in the form of wireframes. The marked image and the prompt word are input into MLLM together, and MLLM returns the analysis results.

[0076] The user prompt for the second step consists of the step task, screenshot description, step rules, and output format. The web page image after the operation is executed is cropped according to the extended coordinate area, and the executed web page elements are marked. The marked image and the prompt are input into MLLM together, and MLLM determines whether additional operations are required. The step task requires judging whether the operation purpose is achieved and whether additional operations are required based on the regional web page screenshots before and after the operation.

[0077] Therefore, in some embodiments, the first inspection module is specifically used to:

[0078] The web page image before the current operation is executed is cropped according to the extended coordinate area, and the web page elements to be executed are marked to obtain a regional web page screenshot before the execution;

[0079] The web page image after the current operation is executed is cropped according to the extended coordinate area, and the executed web page elements are marked to obtain a screenshot of the regional web page after the execution.

[0080] Based on the screenshots of the regional web pages before and after execution, MLLM is used to determine whether the purpose of the current operation has been completed, whether additional operations are required, and if so, a description of the additional operations required.

[0081] Step rules require: 1) If a drop-down window appears for the operation, it is determined that additional operations are required; 2) If the operation is a text input operation, determine whether the input content is correct; 3) The image only shows the content of a certain area of ​​the webpage, and do not consider the impact of the operation on the layout outside the screenshot. The output format requires the output of whether additional operations are required and the description of the additional operations required in the form of a dictionary.

[0082] Through two steps, the changes of the local area webpage before and after the operation are compared, so that MLLM can better judge whether the current operation purpose has been achieved. Here, regional screenshots can be used instead of global images to obtain more accurate results, but it will also increase the time consumption of MLLM services.

[0083] C. A second checking module is used to check whether the user demand is not completed according to the result output by the first checking module and the historical operation.

[0084] The second checking module is used to "check task completion" to control the end of the web page navigation agent operation process.

[0085] Therefore, in some embodiments, the second checking module is specifically used to: if additional operation is required, determine that the user demand is not completed, add the description of the additional operation required to the execution status of the historical operation, and continue to use the perception model module to predict the next operation;

[0086] If no additional operation is required, determine whether the user demand is completed based on historical operations; specifically include: summarizing all web page images after each step of the historical operations, marking the position of the executed web page elements with wireframes on each web page image, and obtaining screenshots of the web page after all operations; determine whether the user demand is completed based on historical operations and the web page screenshots after all operations, as well as the unfinished content if the user demand is not completed.

[0087] If the user's needs are completed, the web page navigation process exits;

[0088] If the user's needs are not completed, the unfinished content will be added to the execution status of the historical operations, and the perception model module will be used to predict the next operation.

[0089] The text prompts of MLLM in this module consist of task description, user requirements, historical operations and output format. The task description requires MLLM to judge whether the user requirements have been met based on historical operations and all screenshots. According to the output format, MLLM will directly output whether the task is completed and unfinished.

[0090] Embodiment 2: Based on Embodiment 1, this embodiment provides a webpage navigation method based on image tags, such as Figure 1 As shown, including:

[0091] Obtain user needs, and execute loop steps until the user needs have been met; wherein the loop steps include:

[0092] S1. If the user's needs are not met, predict the next operation based on the user's needs, historical operations and web page information;

[0093] S2. Based on the screenshots of the regional webpage before and after the current operation, use MLLM to determine whether the operation purpose of the current operation has been completed and whether additional operations are required;

[0094] S3. Check whether the user's needs are not completed based on whether additional operations are required and historical operations;

[0095] If additional operations are required, it is determined that the user's needs are not completed, and the description of the additional operations required is added to the execution status of the historical operations, and the process returns to S1 to continue predicting the next operation;

[0096] If no additional operation is required, determine whether the user's needs are completed based on historical operations; if the user's needs are completed, the web page navigation process exits; if the user's needs are not completed, the unfinished content is added to the execution status of historical operations, and return to S1 to continue predicting the next operation.

[0097] In summary, the web page navigation system and method based on image tagging provided in the embodiments of the present application reduce the content of the DOM tree by marking web page UI images, and at the same time improve the MLLM's understanding of web page operations from different stages, allowing the intelligent agent to predict web page operations more efficiently and accurately.

[0098] Embodiment 3: Based on Embodiment 2, this embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Embodiment 2 are implemented.

[0099] Embodiment 4: Based on embodiment 2, as Figure 3 As shown, this embodiment provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in Example 2 when executing the computer program.

[0100] Embodiment 5: Based on Embodiment 2, this embodiment proposes a hardware system that applies a webpage navigation method based on image tags in Embodiment 2. The method is applied on the server side.

[0101] The terminal may be an electronic device with network access capability, for example, a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a shopping guide terminal, a television, etc.

[0102] The server can be an electronic device with certain computing and processing capabilities. For example, the server can be a server of a distributed system, or a system with multiple processors, memories, network communication modules, etc. operating in collaboration. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server can also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server can also be a new technical means that can realize the corresponding functions of the implementation method of the specification. For example, it can be a new form of "server" based on quantum computing.

[0103] Embodiment 6: Based on Embodiment 2, this embodiment further provides a computer program product comprising instructions, which, when executed by a computer, enables the computer to execute a webpage navigation method based on image tags in Embodiment 2.

[0104] It should be understood that the specific examples in this article are only intended to help those skilled in the art better understand the implementation methods of this specification, rather than limiting the scope of this application.

[0105] It can be understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the implementation methods of this specification.

[0106] It can be understood that the various embodiments described in this specification can be implemented individually or in combination, and the embodiments of this specification are not limited to this.

[0107] Unless otherwise specified, all technical and scientific terms used in the embodiments of this specification have the same meaning as those generally understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in the embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0108] It can be understood that the processor of the embodiment of this specification can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method implementation can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiment of this specification can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of this specification can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0109] It is understood that the memory in the embodiments of this specification may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (programmable ROM, PROM), an erasable programmable read-only memory (erasablePROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0110] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0111] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0112] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0113] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0114] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0115] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0116] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A webpage navigation system based on image tags, characterized in that: include: The perception model module is used to predict the next operation based on user needs, historical operations and web page information; The first checking module is used to: use MLLM to determine whether the operation purpose of the current operation is completed and whether additional operations are required based on the screenshots of the regional webpage before and after the current operation is executed; The second checking module is used to check whether the user demand is not completed according to the result output by the first checking module and the historical operation.

2. The webpage navigation system based on image tagging according to claim 1, characterized in that: The historical operations provide predicted and executed web page operations, each of which includes a web page element DOM, an operation type, an operation purpose, and an execution status; The web page information includes the current web page image and its web page element DOM; The execution status includes a description of the required additional operations and / or unfinished content in the user requirements.

3. The webpage navigation system based on image tagging according to claim 2, characterized in that: The perception model module is specifically used for: If the user's needs are not met; Obtaining position information of the web page element in the previous operation step, and marking the operation element with a wireframe on the web page image in the previous operation step according to the position information to obtain a first image; Fading the current web page image, and adding horizontal and vertical coordinate axes to the faded image to obtain a second image; Based on the first image and the second image, the first multimodal large model MLLM is used to locate and obtain the coordinate information of the layout area where the next operation web page element is located, and expand it N times to obtain an extended coordinate area; Obtain DOM information of all interactive web page elements in the extended coordinate area; obtain coordinate information of web page elements in the extended coordinate area, and draw a position box of each web page element on the web page image according to the coordinates; According to the DOM information of each web page element, mark the position box of the corresponding web page element to obtain a web page screenshot of the marked area; According to the marked area web page screenshot, MLLM is used to determine the current operation web page element DOM, operation type and operation purpose from the DOM information of the web page element.

4. The webpage navigation system based on image tagging according to claim 3, characterized in that: The intervals between the horizontal and vertical coordinate axes are 200 pixels and 100 pixels respectively, and the two coordinate axes are located at the midpoints of the image in the horizontal and vertical directions respectively.

5. The webpage navigation system based on image tagging according to claim 3, characterized in that: Expand 2~3 times to get the extended coordinate area.

6. The webpage navigation system based on image tagging according to claim 3, characterized in that: The first inspection module is specifically used to: The web page image before the current operation is executed is cropped according to the extended coordinate area, and the web page elements to be executed are marked in the form of a wireframe to obtain a regional web page screenshot before the execution; The web page image after the current operation is executed is cropped according to the extended coordinate area, and the executed web page elements are marked in the form of a wireframe to obtain a screenshot of the regional web page after the execution; Based on the screenshots of the regional web pages before and after execution, the second multimodal large model MLLM is used to determine whether the operation purpose of the current operation is completed, whether additional operations are required, and if additional operations are required, the description of the additional operations is provided.

7. The webpage navigation system based on image tagging according to claim 6, characterized in that: The step rule requirements of the second multimodal large model include: 1) if a drop-down window appears for the operation, it is determined that additional operations are required; 2) if the operation is a text input operation, whether the input content is correct is determined; 3) the image only shows the content of a partial area of ​​the web page, and does not consider the impact of the operation on the layout outside the screenshot.

8. The webpage navigation system based on image tagging according to claim 1, characterized in that: The second inspection module is specifically used to: If additional operations are required, it is determined that the user's needs are not completed, and the description of the additional operations is added to the execution status of the historical operations, and the perception model module is returned to continue predicting the next operation; If no additional operation is required, determine whether the user's needs are met based on historical operations; If the user's needs are completed, the web page navigation process exits; If the user's needs are not completed, the unfinished content will be added to the execution status of the historical operations, and the perception model module will be used to predict the next operation.

9. The webpage navigation system based on image tagging according to claim 8, characterized in that: Determine whether user requirements are completed based on historical operations, including: Summarize all web page images after each operation in the historical operation, mark the position of the executed web page elements with a wireframe on each web page image, and obtain the web page screenshots after all operations; Based on historical operations and screenshots of web pages after all operations, determine whether user needs are completed, as well as the unfinished content if user needs are not completed.

10. A webpage navigation method based on image tagging, characterized in that: The method comprises: Obtain user needs, and execute loop steps until the user needs have been met; wherein the loop steps include: S1. If the user's needs are not met, predict the next operation based on the user's needs, historical operations and web page information; S2. Based on the screenshots of the regional webpage before and after the current operation, use MLLM to determine whether the operation purpose of the current operation has been completed and whether additional operations are required; S3. Check whether the user's needs are not completed based on whether additional operations are required and historical operations; If additional operations are required, it is determined that the user's needs are not completed, and the description of the additional operations required is added to the execution status of the historical operations, and the process returns to S1 to continue predicting the next operation; If no additional operation is required, determine whether the user's needs are completed based on historical operations; if the user's needs are completed, the web page navigation process exits; if the user's needs are not completed, the unfinished content is added to the execution status of historical operations, and return to S1 to continue predicting the next operation.

Images