Method for enhancing webpage navigation by using RAG technology and related device

Abstract

The invention discloses a method for enhancing webpage navigation by using an RAG technology and a related device, and the method comprises the steps: obtaining a user demand, and executing a circulation step until the user demand is completed; the circulation step comprises the steps that according to user requirements, webpage screenshots and historical operation, an operation sequence is generated through a first multi-mode large language model; selecting the first operation sequence as a current operation demand, and extracting webpage elements related to the current operation demand in a webpage HTML (Hypertext Markup Language) by utilizing a retrieval enhancement generation (RAG) technology as a retrieval result; according to the current operation demand, the retrieval result and the webpage screenshot, predicting a webpage element needing to be operated currently and an operation type; and obtaining an execution result obtained by executing the webpage element needing to be operated currently according to the corresponding operation type, and updating the webpage screenshot and the historical operation according to the execution result. And webpage elements related to user requirements are retrieved by utilizing an RAG technology, so that the reliability of an MLLM generation result and webpage navigation is improved.

Description

Technical Field

[0001] The present application relates to a method and related device for enhancing web page navigation using RAG technology, and belongs to the field of software technology. Background Art

[0002] Web services are part of people's lives. People can use various websites to search for information, shop, book flights, and other activities. Sometimes, when faced with an uncommon website, people don't know how to operate the web page to achieve their goals, and it takes a certain amount of time to figure out how to use the website. For groups such as the elderly who are not familiar with computer operations, it is even more difficult to use website services, so many websites provide navigation services to guide users' web page operations.

[0003] Traditional web navigation services can only guide routine business operations, but it is difficult to guide operations with special needs of users. Multimodal large models MLLM, such as GPT4o, Claude3.5, Gork2, etc., have strong understanding capabilities and can explore web page operations according to user needs. The intelligent experience developed based on MLLM will help users plan the operation sequence on the web page and perform related operations on their behalf.

[0004] In most cases, in order for MLLM to perceive page information, the HTML of the web page needs to be input into MLLM as part of the prompt words. However, the HTML content of many web pages is too large, and the input limit of the MLLM service cannot withstand excessive text input. Therefore, the mainstream web navigation MLLM agent uses the DOM tree structure to input web page content. The element information in the DOM tree consists of a small number of features, usually only containing type tags and text content. However, the representation method of the DOM tree will lose some important element features, such as class and herf features. In addition, the expression of the DOM tree may exceed the input limit of MLLM; even if it does not exceed the limit, the DOM tree will result in tens of thousands of text content, resulting in high MLLM service fees.

[0005] When the amount of text in the prompt word is too large, MLLM is likely to have hallucination problems or get lost in a large amount of data. When a certain type of input text is long, MLLM tends to only focus on the text at the beginning and end and ignore the text in the middle. In addition, too long prompt words will seriously slow down the reasoning time of MLLM and slow down the response of MLLM services. Summary of the invention

[0006] Retrieval-augmented generation (RAG) technology has been widely used in MLLM agents in recent years. RAG retrieves relevant information based on demand and puts the retrieved information into prompt words to improve the accuracy of MLLM generation results.

[0007] Purpose: In view of at least one of the above technical problems, the present application provides a method and related device for enhancing web page navigation using retrieval enhancement generation RAG technology, which can improve the reliability of web page navigation.

[0008] Technical solution: To solve the above technical problems, the technical solution adopted in this application is:

[0009] In a first aspect, the present application provides a method for enhancing web page navigation using RAG technology, the method comprising:

[0010] Obtain user needs and execute loop steps until the user needs are met; wherein the loop steps include:

[0011] S1. Generate an operation sequence using a first multimodal large language model MLLM according to user needs, web page screenshots and historical operations; the historical operations provide predicted and executed web page operations, each of which includes a web page element, an operation type and an execution result; the initial historical operations are empty;

[0012] S2, selecting the first item of the operation sequence as the current operation requirement, and using the search enhancement generation RAG technology to extract web page elements related to the current operation requirement in the web page HTML as the search result;

[0013] S3, predicting the web page elements and operation types that need to be operated currently based on the current operation requirements, search results and web page screenshots;

[0014] S4. Obtain the execution result obtained by executing the web page element that currently needs to be operated according to the corresponding operation type, and update the web page screenshot and historical operation according to the execution result.

[0015] In a second aspect, the present application provides a device for enhancing web page navigation using RAG technology, the device comprising:

[0016] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0017] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0018] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0019] Beneficial effects: The method and related devices for enhancing web page navigation using RAG technology provided in the present application have the following advantages: using RAG technology to retrieve web page elements related to user needs, allowing MLLM to plan operation elements according to the search results, no longer using the global web page HTML or DOM tree, greatly reducing the input of prompt words, and improving the reliability of MLLM generation results and web page navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of an application scenario of a method according to an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a method flow for enhancing web page navigation using RAG technology according to an embodiment of the present application;

[0022] Figure 3 A schematic diagram of extracting and vectorizing Chinese, English and Pinyin fields according to current operation requirements in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of quantizing text content and feature vectors of each element in a web page element according to an embodiment of the present application;

[0024] Figure 5 A schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] The present application is further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and cannot be used to limit the protection scope of the present application.

[0026] In this application, the professional terms are explained as follows:

[0027] Multimodal Large Language Model (MLLM): Using a powerful large language model (LLM) as the "brain", various multimodal tasks can be performed. The large language model LLM is a deep learning model trained with a large amount of text data, with billions of parameters, capable of understanding and processing natural language. It is a general natural language processing (NLP) model.

[0028] GPT-4o is a language model released by OpenAI for the chatbot ChatGPT, which "can reason about audio, vision, and text in real time.

[0029] Prompt words: text content used to prompt MLLM to provide targeted answers, usually including system prompts used to prompt MLLM in what role, in what scenario, and in what format to generate answers in the current context window; user prompts generally refer to the conversation content entered by the user, used to prompt MLLM to complete tasks, including but not limited to daily conversations, answering questions, translation, writing, and coding.

[0030] Robotic process automation (RPA) is a program that simulates the manual operations of users on the computer desktop through RPA process code to achieve process automation.

[0031] Before describing the embodiments of the present application in detail, a specific scenario example is used to reveal the scenario in which the technical solution of the present application can be used. In a possible scenario example, specifically, Figure 1 As shown, the user performs a specific operation on the client 110, and the client 110 is deployed with a data collection device and a multimodal large language model MLLM, and the data collection device is used to collect web page screenshots and user needs on the client 110; the collected web page screenshots and user needs are transmitted to the server through the network, and the server generates an operation sequence using the first multimodal large language model MLLM according to the web page screenshots, user needs and historical operations through a preset program; the first item of the operation sequence is selected as the current operation demand, and a loop step is executed: the web page elements related to the current operation demand in the web page HTML are extracted as the search result using the retrieval enhancement generation RAG technology; the web page elements and operation types that currently need to be operated are predicted according to the current operation demand, the search result and the web page screenshot; the execution result obtained by executing the web page element that currently needs to be operated according to the corresponding operation type, and the web page screenshot and historical operation are updated according to the execution result; the above steps are repeated until the user demand is completed to realize web page navigation.

[0032] The multimodal large language model MLLM can run on the client terminal side or the server side. The method of the present application can be configured in the multimodal large language model MLLM on the client terminal / or server side, and the client terminal and the server are connected through network transmission to realize multi-terminal interaction.

[0033] The server may be an electronic device with certain computing and processing capabilities. For example, the server may be a server of a distributed system, or a system with multiple processors, memories, network communication modules, etc. operating in collaboration. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server may also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server may also be a new technical means capable of realizing the corresponding functions of the embodiments of the specification. For example, it may be a new form of "server" based on quantum computing.

[0034] The client terminal may be an electronic device with network access capability. Specifically, for example, the terminal may be a desktop computer, a tablet computer, a laptop computer, a smart phone, etc. Alternatively, the terminal may also be software that can run in the electronic device.

[0035] The network may be any type of network that can support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. The one or more networks may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0036] Example 1: Figure 2 As shown, this embodiment provides a method for enhancing web page navigation using RAG technology, which can be applied to a multimodal large language model MLLM, which can run in a client terminal. The method for enhancing web page navigation using RAG technology includes:

[0037] Obtain user needs and execute loop steps until the user needs are met; wherein the loop steps include:

[0038] Step S1, generating an operation sequence using a first multimodal large language model MLLM according to user needs, webpage screenshots and historical operations;

[0039] It should be noted that the historical operations provide predicted and executed web page operations, each of which includes a web page element, an operation type, and an execution result; the initial historical operations are empty.

[0040] In the past, the large language model LLM did not accept image input and could only predict the next operation object based on the HTML or DOM information of the web page, so it could not avoid high-volume prompt words. S1 combines images with text prompt words and uses MLLM to plan the operation sequence; image input only consumes a fixed number of prompt words, such as 1,000 characters, while text prompt words do not exceed 100. For most website scenarios, the general MLLM can plan the operation sequence well, but more special private websites require fine-tuning and training of MLLM.

[0041] Use third-party web automation tools, such as Playwright and Selenium, to obtain web page screenshots; in order to reduce the time for MLLM to process images, scale the web page screenshots to a resolution of about 960*960. Write text prompts, where the text prompts require the first multimodal large language model MLLM to plan an operation sequence related to user needs based on the current web page screenshots and historical operations.

[0042] According to the webpage screenshot and prompt words, MLLM can output the executable operation sequence of the current webpage; for example, if the user demand is "book a flight from Shanghai to Beijing", MLLM plans the operation sequence of "1. Enter the departure place Shanghai, 2. Enter the destination Beijing, 3. Click search". These operation sequences only split the operation logic in the user demand and do not point to specific operation elements, because MLLM currently does not receive the specific webpage content HTML or DOM information.

[0043] S2, selecting the first item of the operation sequence generated in S1 as the current operation requirement, and using the search enhancement generation RAG technology to extract web page elements related to the current operation requirement in the web page HTML as the search result;

[0044] In order to improve the search effect, it is necessary to reduce the difference between the operation requirements and the expression of the HTML information of the web page; for example, if the operation requirement is "click operation", and the corresponding web page element HTML " <span class＝"hs”> search ", the texts of the two are quite different, and the corresponding text vectors have low similarity.

[0045] The HTML of a web page element may indicate the function of a web page element through text content or element features, such as class and title. For Chinese websites, the visible text of the element will be displayed in the most direct Chinese field, such as "search". The representation in the element feature may use pinyin or English, such as "class = search hotel". For web page elements without text content, retrieval and matching can only be performed through the information of the element feature.

[0046] Therefore, in order to effectively retrieve various web page elements, in this embodiment:

[0047] S21, using the second multimodal large language model MLLM to extract the text content in the current operation requirement, wherein the text content includes Chinese fields, English fields and Pinyin fields, such as Figure 3 As shown;

[0048] The extraction rules set in the prompt words are: 1) extract the complete noun description in the operation requirement as the Chinese field; 2) translate the Chinese field into the corresponding English field; 3) obtain the pinyin corresponding to the Chinese field, without the need for phonetic symbols, to obtain the pinyin field. These three types of fields are obtained through MLLM.

[0049] S22, filtering web page elements and element features in the HTML of the web page to obtain the filtered web page elements;

[0050] The vector database in traditional RAG is generated based on a fixed expectation library. The vectorization process of this data is performed before the retrieval service and does not take up the retrieval time. The web page HTML here corresponds to the corpus and is a real-time data, so the construction of the vector database is counted in the retrieval service. In order to reduce the time occupied by the vectorization process, first filter the web page elements according to the element type; the default retained element types are "span", "a", "img", "svg", "input", "textarea", 'label', 'p', 'i', 'button', and these element types have high interactivity. For the two types of "div" and "li", only these types of elements are retained if they contain text.

[0051] For a single web page element, not all element features have search value. Similarly, only fixed element features are retained here, including "title", "aria-label", "id", "herf", "class", "label", "placeholder", and "type". These features are usually used to describe the label, function, and category of an element and have high search value.

[0052] This step S22 reduces the web page elements that need to be retrieved in the web page HTML, and further reduces the available element features of these web page elements.

[0053] After the data processing of S21 and S22, the processed data is vectorized through the natural language NLP model. This method selects a text vector model specifically for RAG from open source resources, which supports the vectorization of Chinese and English.

[0054] S23, such as Figure 3As shown, the NLP vectorization model (the text vector model is used in this embodiment) is used to vectorize the Chinese field, English field, and Pinyin field to obtain the demand-Chinese field vector Q z , demand-English field vector Q e , demand-pinyin field vector Q p ;like Figure 4 As shown, the NLP vectorization model (the text vector model is used in this embodiment) is used to vectorize the text content of the web page elements and the fields of each element feature to obtain the web page element text content vector C t , element eigenvector C i , where i represents the i-th element feature.

[0055] S24. According to the requirements - Chinese field vector Q z , demand-English field vector Q e , demand-pinyin field vector Q p And the web page element text content vector C t , element eigenvector C i , calculate the similarity between the current operation requirement and each web page element; the specific calculation method includes:

[0056] According to demand - Chinese field vector Q z Web page elements text content vector C t Calculate the similarity between the current operation requirement and the visual text of the web page element vt :

[0057] Similarity vt =Q z @C t .T;

[0058] Among them, T means transposing the vector matrix; @ means matrix multiplication;

[0059] Computational Requirements-Chinese Field Vector Q z With each element eigenvector C i Similarity e :

[0060] Similarity z =max(∑ i Q z @C i .T);

[0061] Computational requirements - English field vector Q e With each element eigenvector C i Similarity e :

[0062] Similarity e =max(∑ i Q e @C i .T);

[0063] Computational requirements - Pinyin field vector Q p With each element eigenvector C i Similarity e :

[0064] Similarity p =max(∑ i Q q @C i .T);

[0065] Generally speaking, only one element feature contains useful field information, so the maximum value of all feature vector similarities is selected as the result.

[0066] Calculate element feature similarity attr :

[0067] Similarity attr =max(Similarity z ,Similarity e, Similarity p );

[0068] The maximum value of the similarity of the three element features is selected as the final result, because only one of Chinese, English, and Pinyin can be used to represent the element information in the element feature.

[0069] Finally, calculate the similarity between the current operation requirement and the web page element f :

[0070] Similarity f =0.8*Similarity vt +0.5*Similarity attr ;

[0071] The weight of visual text similarity is set to 0.8, because visual text represents the most intuitive information of web page elements and is relatively important. The weight of element feature similarity is 0.5.

[0072] According to the calculation method of S24, the similarity between the current operation requirement and all the screened web page elements can be calculated.

[0073] S25. Sort the similarities between the current operation requirement and all the screened web page elements from large to small, and select the first K web page elements as the search results.

[0074] In some embodiments, K is 20. The web page elements in these search results are often directly related to the operation requirements, or semantically potentially related elements.

[0075] The search results obtained in step S2 significantly reduce the content of the HTML of the web page. Here, the web page elements in the search results are used to predict specific operations.

[0076] S3, predicting the web page elements and operation types that need to be operated currently based on the current operation requirements, search results and web page screenshots;

[0077] In some embodiments, the position of each web page element in the search results is obtained through the interface of a third-party library such as Playwright, and an image processing library such as Opencv is used to mark the position of these web page elements on the web page screenshot to obtain a marked image; the marked image is sent to the MLLM. The prompt word requires the MLLM to predict the web page element and operation type that need to be operated based on the current operation requirements and the marked image.

[0078] Because S2's retrieval reduces the content of HTML, MLLM can predict results more accurately and faster.

[0079] S4. Obtain the execution result obtained by executing the web page element that currently needs to be operated according to the corresponding operation type, and update the web page screenshot and historical operation according to the execution result.

[0080] It should be noted that the execution of the web page element currently requiring operation in step S4 is automatically executed through a third-party web page automation tool, such as Playwright or Selenium.

[0081] Then, the process returns to repeat S1-S4 to achieve the purpose of planning the operation sequence according to user needs and automatically executing it.

[0082] To sum up, the method of enhancing web page navigation using RAG technology provided in the above implementation uses RAG technology to retrieve web page elements related to user needs, allowing MLLM to plan operation elements according to the search results, no longer using the global web page HTML or DOM tree, greatly reducing the input of prompt words, and improving the reliability of MLLM generation results and web page navigation.

[0083] Embodiment 2: Based on Embodiment 1, this embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Embodiment 1 are implemented.

[0084] Example 3: Based on Example 1, Figure 5 As shown, this embodiment provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in Embodiment 1 when executing the computer program.

[0085] Embodiment 4: Based on Embodiment 1, this embodiment proposes a hardware system that applies the method for enhancing webpage navigation using RAG technology in Embodiment 1. The method is applied on the server side.

[0086] The terminal may be an electronic device with network access capability, for example, a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a shopping guide terminal, a television, etc.

[0087] The server can be an electronic device with certain computing and processing capabilities. For example, the server can be a server of a distributed system, or a system with multiple processors, memories, network communication modules, etc. operating in collaboration. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server can also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server can also be a new technical means that can realize the corresponding functions of the implementation method of the specification. For example, it can be a new form of "server" based on quantum computing.

[0088] Embodiment 5: Based on Embodiment 1, this embodiment further provides a computer program product comprising instructions, which, when executed by a computer, enables the computer to execute a method of enhancing web page navigation using RAG technology in Embodiment 1.

[0089] It should be understood that the specific examples in this article are only intended to help those skilled in the art better understand the implementation methods of this specification, rather than limiting the scope of this application.

[0090] It can be understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the implementation methods of this specification.

[0091] It can be understood that the various embodiments described in this specification can be implemented individually or in combination, and the embodiments of this specification are not limited to this.

[0092] Unless otherwise specified, all technical and scientific terms used in the embodiments of this specification have the same meaning as those generally understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in the embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0093] It can be understood that the processor of the embodiment of this specification can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method implementation can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiment of this specification can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of this specification can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0094] It is understood that the memory in the embodiments of this specification may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (programmable ROM, PROM), an erasable programmable read-only memory (erasablePROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0095] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0097] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0098] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0099] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0100] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0101] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for enhancing web page navigation using RAG technology, characterized in that: The method comprises: Obtain user needs and execute loop steps until the user needs are met; wherein the loop steps include: S1. Generate an operation sequence using a first multimodal large language model MLLM according to user needs, web page screenshots and historical operations; the historical operations provide predicted and executed web page operations, each of which includes a web page element, an operation type and an execution result; the initial historical operations are empty; S2, selecting the first item of the operation sequence as the current operation requirement, and using the search enhancement generation RAG technology to extract web page elements related to the current operation requirement in the web page HTML as the search result; S3, predicting the web page elements and operation types that need to be operated currently based on the current operation requirements, search results and web page screenshots; S4. Obtain the execution result obtained by executing the web page element currently requiring operation according to the corresponding operation type, and update the web page screenshot and historical operation according to the execution result.

2. The method for enhancing web page navigation using RAG technology according to claim 1, characterized in that: The text prompt words of the first multimodal large language model MLLM require the first multimodal large language model MLLM to plan an operation sequence related to user needs based on web page screenshots and historical operations.

3. The method for enhancing web page navigation using RAG technology according to claim 1, characterized in that: Step S2 includes: S21, using the second multimodal large language model MLLM to extract text content in the current operation requirement, wherein the text content includes a Chinese field, an English field, and a Pinyin field; S22, filtering web page elements and element features in the HTML of the web page to obtain filtered web page elements; S23. Use the NLP vectorization model to vectorize the Chinese field, English field, and Pinyin field to obtain the demand-Chinese field vector Q z , demand-English field vector Q e , demand-pinyin field vector Q p , the text content of the web page elements and the fields of each element feature are vectorized to obtain the web page element text content vector C t , element eigenvector C i ; S24. According to the requirements - Chinese field vector Q z , demand-English field vector Q e , demand-pinyin field vector Q p And the web page element text content vector C t , element eigenvector C i , calculate the similarity between the current operation requirement and each web page element; S25. Sort the similarities between the current operation requirement and all the screened web page elements from large to small, and select the first K web page elements as the search results.

4. The method for enhancing web page navigation using RAG technology according to claim 3, characterized in that: S21. Using the second multimodal large language model MLLM to extract the text content in the current operation requirement, specifically including: 1) Extract the complete noun description in the operation requirement as the Chinese field; 2) Translate the Chinese fields into corresponding English fields; 3) Get the pinyin corresponding to the Chinese field, without phonetic symbols, and get the pinyin field.

5. The method for enhancing web page navigation using RAG technology according to claim 3, characterized in that: S22, filtering web page elements and element features in the web page HTML to obtain filtered web page elements, specifically including: First, filter the web page elements by element type; the retained element types are "span", "a", "img", "svg", "input", "textarea", 'label', 'p', 'i', 'button', and "div" and "li" when they contain text; Then, the element features are filtered for each web page element respectively; the retained element features include "title", "aria-label", "id", "herf", "class", "label", "placeholder", and "type".

6. The method for enhancing web page navigation using RAG technology according to claim 1, characterized in that: In S24, the similarity between the current operation requirement and each web page element is calculated, including: According to demand - Chinese field vector Q z Web page elements text content vector C t Calculate the similarity between the current operation requirement and the visual text of the web page element vt : Similarity vt =Q z @C t .T; Among them, T means transposing the vector matrix; @ means matrix multiplication; Computational Requirements-Chinese Field Vector Q z With each element eigenvector C i Similarity e : Similarity z =max(∑ i Q z @C i .T); Computational requirements - English field vector Q e With each element eigenvector C i Similarity e : Similarity e =max(∑ i Q e @C i .T); Computational requirements - Pinyin field vector Q p With each element eigenvector C i Similarity e : Similarity p =max(∑ i Q q @C i .T); Calculate element feature similarity attr : Similarity attr =max(Similarity z ,Similarity e, Similarity p ); Calculate the similarity between the current operation requirement and the web page element f : Similarity f =0.8*Similarity vt +0.5*Similarity attr 。 7. The method for enhancing web page navigation using RAG technology according to claim 3, characterized in that: K is 20.

8. The method for enhancing web page navigation using RAG technology according to claim 1, characterized in that: S3, predicting the web page elements and operation types that need to be operated currently based on the current operation requirements, search results and web page screenshots, specifically including: Obtain the position of each web page element in the search results, mark the position of these web page elements on the web page screenshot with element boxes, and obtain the marked image; According to the current operation requirements and the marked pictures, the third multimodal large language model MLLM is used to predict the web page elements and operation types that need to be operated.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images