Large language model enhancement method, device and equipment based on Internet search

Abstract

The embodiment of the invention provides a big language model enhancement method, device and equipment based on Internet search, and the method comprises the steps: responding to a search request of a user, and obtaining retrieval result data; constructing a first block tree based on the retrieval result data, and performing first pruning operation on the first block tree based on an embedded model to obtain first retrieval refined data; and constructing a second block tree based on the first retrieval refined data, and performing a second pruning operation on the second block tree based on the path generation model to obtain second retrieval refined data, the second retrieval refined data being used for inputting the large language model to generate an answer conforming to the search request. In the embodiment of the invention, the refining speed of the embedded model is relatively high, the refining granularity of the path generation model is finer, and the retrieval result data is refined by combining the embedded model and the path generation model, so that the retrieval refined data with relatively high quality can be obtained, and the answer quality of the large language model is further improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and equipment for enhancing a large language model based on Internet search. Background Art

[0002] Web search engines are one of the main ways for users to actively obtain information from the Internet. Traditional search engines passively accept user search requests and then display a ranked list of search results. In this mode, the interaction between search engines and users is weak, and users need to identify, extract and summarize the effective information in the search results by themselves. In this context, the search enhancement generation technology that combines large language models and information retrieval technology will play an increasingly important role in the future Internet. After being applied to the search enhancement generation technology, the search engine will refine the search results together with the user request and input them into the large language model to generate the final answer.

[0003] Due to the limitation of the input window of the large language model, the step of refining the search results is particularly critical. Existing search result refining algorithms include: slicing-rearrangement and summary generation, etc., and their methods are mainly to optimize the search results from plain text. However, the search results of search engines are mainly in the format of Hypertext Markup Language (HTML). If the HTML text optimization is simplified to plain text optimization, the content and structure information may be lost, and the core content cannot be extracted. The poor effect of the refining algorithm will in turn affect the output quality of the large language model, and the answer generated in the end may not achieve the ideal effect, affecting the user experience. Summary of the invention

[0004] In view of this, the present application provides a large language model enhancement method, device and equipment based on Internet search, so as to solve the problem of poor retrieval result refinement effect in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a large language model enhancement method based on Internet search, comprising:

[0006] Responding to a user's search request, obtaining search result data;

[0007] Constructing a first block tree Block Tree based on the search result data, and performing a first pruning operation on the first block tree based on the embedded model to obtain first search refined data;

[0008] Building a second tree based on the first search refined data, wherein the granularity of the second tree is smaller than that of the first tree;

[0009] A second pruning operation is performed on the second tree block based on the path generation model to obtain second search refined data, where the second search refined data is used to input into a large language model to generate an answer that matches the search request.

[0010] In an optional embodiment, constructing a first block tree based on the search result data includes:

[0011] Processing the search result data based on a document cleaning algorithm;

[0012] Convert the processed search result data into a format to generate a corresponding DOM tree;

[0013] The DOM tree is processed based on a first tree construction algorithm to construct the first tree.

[0014] In an optional embodiment, the processing of the search result data based on a document cleaning algorithm includes:

[0015] Clearing CSS format text and JavaScript scripts in the search result data;

[0016] Clearing the attributes of the Hypertext Markup Language HTML tags in the search result data;

[0017] Merge tags of HTML documents and generate a single HTML document.

[0018] In an optional embodiment, the processing the DOM tree based on the first block tree construction algorithm to construct the first block tree includes:

[0019] Traversing each node of the DOM tree based on a breadth-first algorithm;

[0020] For any node, when it is detected that the total number of words in all the content contained in the current node is less than the preset word count threshold, all the content of the current node is merged, and the merged node content is regarded as a block;

[0021] For any node, when it is detected that the total number of words in all the contents contained in the current node is greater than or equal to the word number threshold, the current node is not operated and the child nodes of the current node are traversed.

[0022] In an optional embodiment, performing a first pruning operation on the first tree block based on the embedded model to obtain first search refined data includes:

[0023] Calculating the relevance of each block in the first block tree to the search request of the user based on the embedded model;

[0024] Performing pruning operations on the blocks of the first block tree in order from small to large based on the correlation;

[0025] The format of the first block of trees that has completed the pruning operation is converted to obtain the first search refined data.

[0026] In an optional embodiment, performing a second pruning operation on the second tree block based on the path generation model to obtain second search refined data includes:

[0027] Calculating, based on the path generation model, a path generation probability of each block in the second block tree under the search request condition of the current user;

[0028] Performing pruning operations on the blocks of the second block tree in sequence based on the order of path generation probability from small to large;

[0029] The format of the second block tree that has completed the pruning operation is converted to obtain the second search refined data.

[0030] In an optional embodiment, the calculating, based on the path generation model, the path generation probability of each block in the second block tree under the search request condition of the current user includes:

[0031] Constructing a token tree corresponding to the second block tree, wherein the token tree comprises a plurality of sequences, and each sequence comprises at least one node;

[0032] Calculate the first probability of each node in the same sequence of the token tree based on the path generation model;

[0033] Each node of the token tree is traversed based on a depth-first algorithm, and for any traversed node, a path generation probability of the current node is determined based on the path of the current node in the token tree and the first probability of each node of the token tree.

[0034] In a second aspect, an embodiment of the present application provides a large language model enhancement device based on Internet search, comprising:

[0035] An acquisition module, used to obtain retrieval result data in response to a user's search request;

[0036] A first processing module, configured to construct a first block tree based on the search result data, and perform a first pruning operation on the first block tree based on an embedded model to obtain first search refined data;

[0037] A second processing module, configured to construct a second block tree based on the first search and refinement data, wherein the granularity of the second block tree is smaller than that of the first block tree;

[0038] The second processing module is further used to perform a second pruning operation on the second block tree based on the path generation model to obtain second search refined data, and the second search refined data is used to input the large language model to generate an answer that matches the search request.

[0039] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute any one of the methods described in the first aspect above.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any method described in the first aspect.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes any method described in the first aspect.

[0042] By adopting the scheme provided in the embodiment of the present application, in response to the user's search request, the search result data is obtained; based on the search result data, a first block tree Block Tree is constructed, and a first pruning operation is performed on the first block tree based on the embedded model to obtain first search refined data; based on the first search refined data, a second block tree is constructed, and a second pruning operation is performed on the second block tree based on the path generation model to obtain second search refined data, and the second search refined data is used to input the large language model to generate an answer that conforms to the search request. In the embodiment of the present application, the embedded model refines faster, and the path generation model refines finer granularity. By combining the embedded model and the path generation model to refine the search result data, high-quality search refined data can be obtained, thereby improving the answer quality of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 A flowchart of a large language model enhancement method based on Internet search provided in an embodiment of the present application;

[0046] Figure 2 An example schematic diagram of another large language model enhancement method based on Internet search provided in an embodiment of the present application;

[0047] Figure 3A An example schematic diagram of another large language model enhancement method based on Internet search provided in an embodiment of the present application;

[0048] Figure 3B An example schematic diagram of another large language model enhancement method based on Internet search provided in an embodiment of the present application;

[0049] Figure 4 An example schematic diagram of another large language model enhancement method based on Internet search provided in an embodiment of the present application;

[0050] Figure 5A An example schematic diagram of another large language model enhancement method based on Internet search provided in an embodiment of the present application;

[0051] Figure 5B An example schematic diagram of another large language model enhancement method based on Internet search provided in an embodiment of the present application;

[0052] Figure 6 An example schematic diagram of another large language model enhancement method based on Internet search provided in an embodiment of the present application;

[0053] Figure 7 An example schematic diagram of another large language model enhancement method based on Internet search provided in an embodiment of the present application;

[0054] Figure 8 A flowchart of another large language model enhancement method based on Internet search provided in an embodiment of the present application;

[0055] Fig. 9 A schematic diagram of the structure of a large language model enhancement device based on Internet search provided in an embodiment of the present application;

[0056] Fig.10A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0058] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0059] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0060] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0061] Nowadays, users can actively obtain relevant information from the Internet through web search engines to solve various problems. Traditional search engines passively accept users' search requests and then display a ranked list of search results. In this mode, the interaction between search engines and users is weak, and users need to distinguish, extract and summarize the effective information in the search results by themselves. With the release of ChatGPT, the Large Language Model (LLM) based on the Transformer decoder architecture has been proven to have strong language ability, knowledge ability and reasoning ability.

[0062] In some embodiments, the GPT model based on the Transformer architecture may also include pre-training, supervised fine-tuning (SFT), and reinforcement learning modules. Among them, the pre-training of a large language model refers to building a large neural network model and feeding a large amount of data to train the language model in some way. The main feature of the pre-training of a large language model is that the amount of data used to train the language model is large enough and the model is large enough. SFT refers to pre-training a neural network model on a source data set, namely the source model. Then a new neural network model is created, namely the target model. The target model copies all model designs and parameters of the source model except the output layer. These model parameters contain the knowledge learned on the source data set, and this knowledge is also applicable to the target data set. The output layer of the source model is closely related to the label of the source data set, so it is not used in the target model. During fine-tuning, an output layer with an output size of the number of categories of the target data set is added to the target model, and the model parameters of the layer are randomly initialized. When training the target model on the target data set, it will be trained from the beginning to the output layer, and the parameters of the remaining layers are fine-tuned based on the parameters of the source model. Reinforcement learning is one of the paradigms and methodologies of machine learning, which is used to describe and solve the problem of how intelligent agents learn strategies to maximize rewards or achieve specific goals during their interactions with the environment. The central idea is to let the agent learn in the environment. Each action will correspond to its own reward, and the agent will learn what to do under what circumstances by analyzing data. It is to solve the problem of sequential decision-making, in which a decision-making agent interacts iteratively with a discrete time dynamic system. At the beginning of each time step, the system is in a certain state. Based on the agent's decision rule, it observes the current state and selects one from a finite set of states. Then, the dynamic system will enter the next new state and obtain a corresponding benefit. This cycle of state selection is repeated to obtain a set of maximized benefits.

[0063] However, large language models have problems such as difficulty in accurately memorizing long-tail knowledge, outdated knowledge, and hallucinations. Therefore, Retrieval Augmented Generation (RAG) technology, which combines large language models with information retrieval technology, has become the main technology for building the next generation of search engines. This technology retrieves relevant information from external knowledge bases and inputs it as prompts to large language models (LLMs) to enhance the ability of large language models to handle knowledge-intensive tasks, such as question answering, text summarization, and content generation. Its main steps include: (1) Retrieval: Retrieve information related to the question from the Internet. The purpose of this step is to provide useful contextual information and knowledge support for the subsequent generation process. (2) Enhancement: Use the retrieved information as context input for the large language model to enhance the large language model's ability to understand and answer specific questions. The purpose of this step is to integrate external knowledge into the generation process, making the generated text content richer, more accurate, and in line with user needs. (3) Generation: Use the retrieved information as context input and combine it with the large language model to generate text content.

[0064] Retrieval enhancement generation is a complex system, which usually includes a retrieval necessity judgement device, a query rewriter, a retriever, a refiner, and a reader. The performance of these components affects the final effect of interaction with the user. Among them, the refiner plays a key role in filtering irrelevant information in the recalled documents and extracting effective information. In the retrieval enhancement generation technology, the retrieval results and user requests are filled into the input window of the large language model together. Limited by the input window size and inference overhead of the large language model, the retrieval content needs to be as short as possible while ensuring that no information is lost. Therefore, the retrieval content needs to be refined before entering the large language model to reduce the volume of the retrieval content. The refinement of the retrieval content is often undertaken by a lightweight model with much smaller parameters than the large language model. Common retrieval result refinement algorithms are as follows: (1) Slicing-rearrangement: First, the retrieval results are divided into slices of similar size according to a certain text structure, and then the text vector model is used to calculate the semantic similarity between the slices and the user request. The slices with high relevance are retained and combined into the refined results. (2) Summary generation: Using the generative model, the summary of the retrieval results is directly generated as the refined result.

[0065] Slicing and reordering is a classic method for optimizing recall results, and its advantages are high efficiency and strong reliability. Many mainstream RAG frameworks use this method, such as the LangChain framework and the LlamaIndex framework. The slicing and reordering method first slices the retrieved documents according to certain rules, and then uses the reranking model to select slices with high relevance. Traditional methods slice based on paragraphs or sentences. For HTML document slicing, LangChain recognizes specific HTML tags (such as titles) as slice markers; while LlamaIndex first converts HTML to plain text, and then uses a general slicing method for plain text. In terms of reranking models, models such as BM25, BGE, and RePLLaMa can be used to complete related operations.

[0066] Summarization refiners usually use a generative language model. Some work uses existing summary models to summarize the recall results in segments and hierarchies. In order to make the refinement results more in line with the preferences of LLM, many methods use large language models to provide supervision signals to train refiners. The supervision method can be to provide supervision signals in reinforcement learning or to generate labeled data using LLM. Similarly, LLM can also be used to supervise the reordering model in the slice-rearrange method.

[0067] The above methods are all based on optimizing retrieval results from plain text. When RAG serves general web search engines, the recall results are mainly in HTML format. If HTML text optimization is simplified to plain text optimization, many problems may occur. First, converting HTML to plain text will lead to the loss of content and structural information. Second, methods such as block reordering may lead to unreasonable slicing because the slicing process does not consider the structure of HTML. In the summary method, the summary model may face problems such as too long text, high computational cost, or limited understanding of HTML, making it unable to extract the core content. In the link of retrieval enhancement generation, the retrieval result refinement step directly affects the quality of the generated answer and the computational overhead of large language model reasoning. Therefore, the performance of retrieval result refinement largely affects the overall performance of retrieval enhancement generation. Existing methods starting from plain text retrieval results cannot adapt well to the HTML format content recalled from the Internet.

[0068] Taking into account the defects of the above technologies, the embodiment of the present application proposes a large language model enhancement method based on Internet search. Taking into account the characteristics of HTML documents recalled by search engines, such as complex content structure, diverse content, low information density, and long text length, while shortening the length of the search results, it retains valid information as much as possible and maintains the HTML structural characteristics.

[0069] Figure 1A flowchart of a large language model enhancement method based on Internet search provided in an embodiment of the present application is provided. The method can be applied to a processor such as Figure 1 As shown, the method may include:

[0070] Step 101, in response to a user's search request, obtaining retrieval result data.

[0071] Step 102: construct a first block tree BlockTree based on the search result data, and perform a first pruning operation on the first block tree based on the embedded model to obtain first search refined data.

[0072] Step 103: construct a second tree based on the first search and refinement data, wherein the granularity of the second tree is smaller than that of the first tree.

[0073] Step 104, performing a second pruning operation on the second tree block based on the path generation model to obtain second search refined data, and the second search refined data is used to input the large language model to generate an answer that matches the search request.

[0074] After the user enters the search request, the processor first searches for relevant search result data, which mainly includes HTML documents obtained from the network. Optionally, the user can manually enter the search request in the input interface of the search engine. Alternatively, the user can also manually or voice input the search request in some application software APP that supports search enhancement generation technology. Other common search scenarios can also apply the method of the embodiment of the present application.

[0075] Considering the output efficiency and quality of LLM, LLM sets a maximum input window L. On the other hand, the total length of the initial HTML document D recalled from the Internet is usually large, exceeding the limit of the LLM input window. The processor needs to map the initial HTML document D to a smaller document d and ensure that the total length of d can fit into the LLM context window, that is, the total length of d must be less than or equal to L. In addition, in order to ensure the output quality of LLM, the mapping from HTML document D to document d should retain valid key information as much as possible. The embodiment of the present application deletes unnecessary information in HTML document D by block tree pruning so that its total length conforms to the input window of LLM.

[0076] In an optional embodiment, before the processor constructs the first block tree BlockTree based on the search result data, it will clean and organize the HTML document based on the document cleaning algorithm to facilitate the construction of the block tree in the subsequent steps. The cleaning and organization of the HTML document mainly includes the following processes:

[0077] (1) Clear the CSS formatting text and JavaScript scripts in the HTML document. CSS formatting text is used to determine the display style of the HTML document, such as the background color or font style of a certain area. JavaScript scripts can be used to add interactive functions to HTML documents.

[0078] (2) Clear the attributes of HTML tags by matching rules. The attributes of HTML tags can be combined with CSS format text to set the display format of HTML documents. For example, when changing the background color of HTML through CSS format text, you can specify the background color corresponding to a certain type of tag attribute.

[0079] (3) Merge multiple layers of single nested HTML tags. When there is only one HTML tag nested under a certain HTML tag, you can merge the HTML tags to simplify the overall structure.

[0080] (4) Merge all HTML documents into one large HTML tag to form one HTML document.

[0081] Through the above overall cleaning steps, the processor removes unnecessary information in the HTML document, simplifies the structure of the HTML document, and obtains a final HTML document.

[0082] In step 102, the processor may first convert the cleaned and organized HTML document into a DOM tree format, and then process the DOM tree based on the first tree construction algorithm to construct the first tree. Specifically, the processor may use an existing HTML parsing tool (such as Beautiful Soup) to parse the HTML document into a Document Object Model (DOM) tree format. The DOM tree provides an access model to the entire HTML document, treating the HTML document as a tree structure, and each node of the tree represents an HTML tag or a text item within a tag. The DOM tree structure accurately describes the interrelationships between tags in an HTML document, and the process of converting an HTML document into a DOM tree is called parsing. After the HTML document is parsed, it is converted into a DOM tree, so the processing of the HTML document can be achieved by operating the DOM tree. Using the methods and properties of the object, the processor can easily access, modify, add and delete the nodes and content of the DOM tree.

[0083] The structure of the DOM tree is relatively fine. If the DOM tree is used directly for pruning, the following problems may occur: (1) A large proportion of nodes in the DOM tree have very little text attached to them (for example, the text attached to some nodes does not exceed 5 words), making it difficult for the embedded model to capture its text features. (2) The DOM tree contains a large number of nodes, which will bring huge computational costs to the embedded model and the path generation model. (3) The depth of the DOM tree is very deep. When the path generation model generates a path, a deeper tree will bring greater computational overhead.

[0084] To address the above issues, the processor can convert the DOM tree into a block tree based on a block tree construction algorithm with adjustable granularity. In the embodiment of the present application, the elements under a node are defined as follows: (1) All elements under the node are called node content node.contents. (2) The nested sub-tags of the current node are called child nodes node.children. (3) The text directly attached to the node is called node text node.text. That is, for any node, its node content includes: child nodes and / or node text. Refer to Figure 2 , the contents in the dotted box are nodes The dotted box indicated by arrow 202 directly contains text, which is called the node text of the current node. The dotted box indicated by arrow 201 first nests other sub-tags (such as <h1>、 or ), and then attach text after the child tag, which is called the child node of the current node.

[0085] To distinguish between sub-elements with the same HTML tag, a number can be appended to the end of the original tag name. For example, two sub-elements with the same " "The sub-elements of the tag are renamed to" <div1>"and" <div2>".

[0086] Based on the above definition, the processor converts the DOM tree into a block tree based on the block tree construction algorithm. In the block tree, the block is the smallest unit and can be pruned in subsequent steps. Specifically, the processor can use the breadth-first algorithm to traverse all nodes in the DOM tree and construct the block tree according to the following rules: (1) The visited leaf nodes are directly regarded as a block. (2) If the total number of tokens of all content under a node is less than the token number threshold maxTokens we set, the processor will merge all the content of the node and regard it as a block. It should be noted that if there are only child nodes under a node but no node text, the node will not be regarded as a block. The number of tokens is used to represent the number of words in the text. The longer the text contained in a node, the larger its number of tokens is usually.

[0087] For the specific process, please refer to Figure 3A In the code shown, in lines 1 to 4, a block tree structure (T) is first declared, the root node of T is assigned to R, and R is pushed into the nodeQueue queue. Lines 5 to 26 are the process of traversing each node of the DOM tree. Specifically, in lines 6 to 9, for any node of the traversed DOM tree, if the current node is a leaf node, the node content of the current node is regarded as a block. In lines 11 to 14, for any node of the traversed DOM tree, if the total number of tokens contained in the content of the current node is less than maxTokens, all node contents of the current node are regarded as a block. In lines 16 to 22, for nodes with a token number greater than or equal to maxTokens, the child nodes of the current node are traversed. If the total number of tokens contained in the content of the child node is less than maxTokens, the node content of the corresponding child node is regarded as a block. If the total token tree of the content contained in the child node is still greater than or equal to maxTokens, the child nodes of the child node are traversed again.

[0088] Through the above process, the block tree construction algorithm merges fragmented nodes into one block until the number of tokens exceeds maxTokens. By adjusting maxTokens, the granularity of the block tree can be adjusted to make the block tree adapt to different pruning algorithms.

[0089] After the first block tree is built, the processor can prune the first block tree based on the embedded model to delete irrelevant blocks while retaining key information as much as possible. Specifically, the processor can calculate the correlation between each block and the search request input by the user based on the embedded model, and then prune the block tree in order from small to large correlation until the total length of the HTML document meets the input window of the LLM. During the calculation, each block and the search request input by the user are represented by a high-dimensional vector (such as 512 dimensions or other optional dimensions), and the embedded model can calculate the correlation between each block and the search request based on the high-dimensional vector.

[0090] When pruning the first block tree, a greedy algorithm can be used to complete the pruning operation. Specifically, when deleting a block, if the block is a leaf node, the block is deleted directly. If the block consists of text directly attached to the parent node, only the node text node.text is deleted. After deleting a block, the processor checks whether the parent node is empty. If the parent node is empty, it is deleted.

[0091] For the specific process, please refer to Figure 3B In the code shown, in lines 1 to 5, the processor pushes all nodes containing blocks in the block tree into the nodes queue, and calculates the semantic similarity node.score (i.e., relevance) between each node in the nodes queue and the user's search request. In line 6, the processor sorts the nodes in the nodes queue in ascending order according to the relevance node.score. In lines 7 to 19, the processor deletes nodes with low relevance through a while loop statement until the length of the HTML document meets the requirements. Specifically, in lines 8 to 11, if the current node being traversed is a leaf node, the current node is deleted directly. In lines 12 to 14, if the parent node of the deleted node does not contain text, the parent node is deleted. In lines 16 to 17, if the current node being traversed is not a leaf node, the current node is not deleted, and only the node text of the current node is deleted.

[0092] The structure of the first tree can be referred to Figure 4 In the example shown, nodes 41, 42, 44, 45, 46, and 47 are all leaf nodes, and are directly regarded as a block. The nodes with gray background will be pruned and deleted. For node 43, there are two child nodes below it: node 46 and node 47. When nodes 46 and 47 are pruned and deleted, since node 43 has no other child nodes and node 43 does not contain node text, node 43 will also be deleted. Nodes 41, 44, 46, and 47 are regular leaf nodes, and can be directly pruned and deleted.

[0093] After completing the pruning operation based on the embedded model, the processor converts the block tree back into an HTML document that satisfies the input window of the LLM. The block tree pruning algorithm based on the embedded model is a lightweight but effective algorithm that is more adaptable to the retrieved HTML than existing refinement algorithms. However, the algorithm still has limitations, which are mainly reflected in the following aspects: (1) The context window of the embedded model is limited to the text range within each block, and the candidate blocks cannot be directly compared within a single inference time. Therefore, the embedded model lacks a global grasp of the document information. (2) The embedded model cannot handle block trees with finer granularity because the text within most blocks is not sufficient for the embedded model to obtain semantic features.

[0094] Taking into account the shortcomings of the block tree pruning algorithm based on the embedded model, the embodiment of the present application also proposes a block tree pruning algorithm based on the path generation model. In step 103, the processor constructs another block tree with finer granularity, namely the second block tree, based on the first search refined data after pruning by the embedded model. When constructing the second block tree, the granularity of the second block tree can be made smaller than that of the first block tree by adjusting maxTokens. In response to the user's search request, the path generation model can output the generation probability of each node in the block tree. The higher the generation probability of the node, the closer the node content is to the best answer to the current search request. Afterwards, the processor prunes the block tree based on the generation probability of the node and deletes nodes with lower generation probabilities.

[0095] Specifically, for each block path, the processor can segment it into tokens: (t1, t2, ...), and calculate the probability P(t n ), and multiply them all together to get the generation probability. When the model calculates the generation probability of the nth token in the output given the inputs and n-1 generated output tokens, Logits(t n )=GenModel(t b |{input,t1,t2,…,t n-1 }).

[0096] In order to simplify some of the intermediate calculation steps, the processor can construct a corresponding token tree Tokentree based on the block tree. Figure 5A After the Token tree is constructed, we calculate the probability of each node in the Token tree. The specific steps include: (1) The probability of the root node is 1.0, usually the root node is "<", which depends on the tokenizer. (2) If a parent node has only one child node, the probability of the child node is 1.0. (3) The probabilities of other nodes are calculated by the path generation model GenModel. The Logits calculation formula described above is used to obtain the probability of each node. The logits of , then the probability of any node in each sequence can be obtained by the following formula:

[0097]

[0098] Among them, n=1 represents the first sequence of the Token tree, that is, "<", as the root node, and its probability is 1.0. K represents the number of child nodes contained in a parent node. When K=1, it means there is only one child node, and the probability of the child node is 1.0. represents the Kth child node of the nth sequence. In the first two cases, there is no need to use a generative model for inference, which means we can skip many tokens.

[0099] The above calculation rules can skip the calculation of the probability of some nodes (root nodes, single child nodes), reducing the calculation cost of the algorithm. For other nodes, the processor can traverse the Tokentree based on the depth-first algorithm and calculate the probability of the node. Under the depth-first algorithm, the nodes calculated successively have the largest token prefix sequence.

[0100] For the specific process, please refer to Figure 5B In the code shown, in lines 2 to 5, the processor sets the probability of the root node to 1.0. In lines 6 to 21, the processor calculates the probabilities of other nodes. Specifically, in lines 9 to 10, if there is no next child node, the next step is not performed. In lines 11 to 13, if there is only one child node, the probability of the child node is set to 1. In lines 14 to 20, for other nodes, the probability of each node is calculated according to the above formula.

[0101] For any node in the second block tree, the generation probability of the node can be obtained by multiplying the probabilities of all nodes on the path to the node. The calculation formula is as follows:

[0102]

[0103] Among them, score(t n ) is the probability of each node in the Token tree, and Score(block) is the generation probability of each node in the second block tree. The processor prunes the second block tree in descending order of generation probability, which can further delete irrelevant information and obtain a more refined HTML document.

[0104] The pruning of the second tree can be referred to Figure 6 In the example shown, nodes 602, 603, 605, 607, 608, 609, 610, and 611 are blocks of the second tree. The nodes with gray backgrounds have a low probability of generation and will be pruned and deleted. After nodes 607 and 608 are deleted, all child nodes of node 606 have been deleted, and node 606 will also be deleted. Similarly, after nodes 605 and 606 are deleted, node 604 will also be deleted, and after nodes 602 and 603 are deleted, node 601 will be deleted.

[0105] In the embodiment of the present application, the computational cost of the embedded model is relatively small, and the path generation model can process nodes with finer granularity. Combining the embedded model and the path generation model can improve the refinement quality of the HTML document, thereby improving the output quality of the large language model.

[0106] In an optional embodiment, in order to enhance the ability of the path generation model, the user can manually select specific data to train the model. First, the user's query is sampled from multiple common open source data sets. For each query, an online search engine is used to retrieve relevant HTML documents. For the HTML documents obtained by the search, Figure 1 Steps 101 to 105 are processed to generate a block tree. For each block, the matching degree between each node text and the best answer is calculated based on the path of the best answer. Figure 7 , during the training process, define the format of the output answer. The best answer corresponding to the current question is "In the historic 2018 Royal Rumble, Shinsuke Nakamura won the men's Royal Rumble...", and the path of the best answer is " <htm14> <div1> <div3> ", this path is the best path for the current problem. After training, in the inference phase, the path generation model can output the generation probability of each node in the block tree. Afterwards, the processor can perform pruning operations on the block tree in order of generation probability from small to large.

[0107] Figure 8 A flowchart of another large language model enhancement method based on Internet search provided in an embodiment of the present application. Figure 8 As shown, the method may include:

[0108] Step 801: In response to a user's search request, search and obtain an HTML document.

[0109] Step 802, cleaning and organizing the HTML document.

[0110] Step 803: parse the HTML document into a DOM tree, and convert the DOM tree into a first block tree.

[0111] Step 804: Perform a first pruning on the first tree based on the embedded model.

[0112] Step 805: construct a second tree with finer granularity based on the HTML document after the first pruning.

[0113] Step 806: Perform a second pruning on the second tree based on the path generation model.

[0114] In the embodiment of the present application, based on the respective characteristics of the embedded model and the path generation model, the processor successively constructs two block trees with different granularities. The granularity of the first block tree is relatively large, and the processor can perform a first pruning on the first block tree based on the embedded model to achieve preliminary refinement of the HTML document, and the overall refining speed is relatively fast. Based on the HTML document that has completed the preliminary refinement, a second block tree can be constructed, and the granularity of the second block tree is relatively small. The processor can perform a second pruning on the second block tree based on the path generation model to achieve more detailed refinement of the HTML document.

[0115] In an optional embodiment, the process of the processor converting the DOM tree into the first block tree may include: (1) traversing each node of the DOM tree based on the breadth-first algorithm; (2) for any node, when it is detected that the total number of words contained in all the contents of the current node is less than a preset word count threshold, merging all the contents of the current node and treating the merged node content as a block; (3) for any node, when it is detected that the total number of words contained in all the contents of the current node is greater than or equal to the word count threshold, not operating the current node and traversing the child nodes of the current node. Based on the above process, the processor can convert the DOM tree into the first block tree. The word count threshold can be regarded as the maxTokens mentioned above, and block trees with different granularities can be constructed by adjusting the word count threshold.

[0116] For the first tree, the processor can calculate the relevance of each block to the user search request based on the embedded model, and perform pruning operations in order of relevance from small to large. After the pruning operation is completed, the first tree is formatted to obtain the first search refined data.

[0117] For the first retrieval refined data after preliminary refinement, the processor can construct a second block tree, and the second block tree has a finer granularity than the first block tree. The processor can calculate the generation probability of each block in the second block tree based on the path generation model, and the generation probability is used to represent the degree of match between the text content of the current block and the best answer. The second retrieval refined data can be obtained by pruning the second block tree based on the generation probability and converting the format of the second block tree that has completed the pruning operation. Specifically, in the process of calculating the generation probability, the processor can construct a Token tree corresponding to the second block tree, and each block is converted into one or more sequences in the Token tree, and each sequence contains at least one node. Afterwards, the processor calculates the first probability of each node in the same sequence of the Token tree. For details, please refer to Figure 6 The processor traverses each node of the Token tree based on the depth-first algorithm. For any traversed node, the generation probability of the current node is obtained by multiplying the first probabilities of each node included in the path where the current node is located.

[0118] In the embodiment of the present application, the embedded model refines at a faster speed, and the path generation model refines at a finer granularity. By combining the embedded model and the path generation model to refine the retrieval result data, higher-quality retrieval refined data can be obtained, thereby improving the answer quality of the large language model.

[0119] Fig. 9 A schematic diagram of the structure of a large language model enhancement device based on Internet search provided in an embodiment of the present application. The device can be deployed with a processor, such as Fig. 9 As shown, the device may include: an acquisition module 910 , a first processing module 920 , and a second processing module 930 .

[0120] The acquisition module 910 is used to acquire retrieval result data in response to a user's search request.

[0121] The first processing module 920 is used to construct a first block tree Block Tree based on the search result data, and perform a first pruning operation on the first block tree based on the embedded model to obtain first search refined data.

[0122] The second processing module 930 is used to construct a second tree based on the first search refined data, and perform a second pruning operation on the second tree based on the path generation model to obtain second search refined data, and the second search refined data is used to input the large language model to generate an answer that matches the search request. The specific process can refer to the description in the above method flowchart.

[0123] Corresponding to the above embodiments, the present application also provides an electronic device. Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device 1000 may include: a processor 1001, a memory 1002 and a communication unit 1003. These components communicate through one or more buses, and those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiment of the present application, it can be a bus structure or a star structure, and can also include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0124] The communication unit 1003 is used to establish a communication channel so that the electronic device can communicate with other devices, receive user data sent by other devices or send user data to other devices.

[0125] The processor 1001 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs, instructions, and / or modules stored in the memory 1002, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 1001 can include only a central processing unit (CPU). In the embodiment of the present application, the CPU can be a single computing core or multiple computing cores.

[0126] The memory 1002 is used to store the execution instructions of the processor 1001. The memory 1002 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0127] When the execution instructions in the memory 1002 are executed by the processor 1001, the electronic device 1000 can execute Figure 1 Some or all of the steps in the illustrated embodiments.

[0128] In a specific implementation, the present application also provides a computer storage medium, wherein the computer storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment of the large language model enhancement method based on Internet search provided by the present application. The storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0129] In a specific implementation, the present application also provides a computer program product, wherein the computer program product includes executable instructions, and when the executable instructions are executed on a computer, the computer executes part or all of the steps in each embodiment of the large language model enhancement method based on Internet search provided in the present application.

[0130] An embodiment of the present application also provides a non-temporary computer-readable storage medium, which stores computer instructions. The computer instructions enable the computer to execute the large language model enhancement method based on Internet search provided in an embodiment of the present application.

[0131] The above-mentioned non-temporary computer-readable storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (Read Only Memory; hereinafter referred to as: ROM), an erasable programmable read-only memory (ErasableProgrammable Read Only Memory; hereinafter referred to as: EPROM) or flash memory, optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0132] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0133] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0134] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0135] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the device embodiment and the terminal embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment. < / div3> < / div1> < / htm14> < / h1>

Claims

1. A large language model enhancement method based on Internet search, characterized in that: include: Responding to a user's search request, obtaining search result data; Constructing a first block tree Block Tree based on the search result data, and performing a first pruning operation on the first block tree based on the embedded model to obtain first search refined data; Building a second block tree based on the first search refined data, wherein the granularity of the second block tree is smaller than that of the first block tree; A second pruning operation is performed on the second tree block based on the path generation model to obtain second search refined data, where the second search refined data is used to input into a large language model to generate an answer that matches the search request.

2. The method according to claim 1, characterized in that: The step of constructing a first block tree based on the search result data includes: Processing the search result data based on a document cleaning algorithm; Convert the processed search result data into a format to generate a corresponding DOM tree; The DOM tree is processed based on a first tree construction algorithm to construct the first tree.

3. The method according to claim 2, characterized in that The processing of the search result data based on the document cleaning algorithm includes: Clearing CSS format text and JavaScript scripts in the search result data; Clearing the attributes of the Hypertext Markup Language HTML tags in the search result data; Merge tags of HTML documents and generate a single HTML document.

4. The method according to claim 2, characterized in that: The step of processing the DOM tree based on the first tree construction algorithm to construct the first tree includes: Traversing each node of the DOM tree based on a breadth-first algorithm; For any node, when it is detected that the total number of words in all the content contained in the current node is less than the preset word count threshold, all the content of the current node is merged, and the merged node content is regarded as a block; For any node, when it is detected that the total number of words in all the contents contained in the current node is greater than or equal to the word number threshold, the current node is not operated and the child nodes of the current node are traversed.

5. The method according to claim 1, characterized in that The step of performing a first pruning operation on the first tree block based on the embedded model to obtain first search refined data includes: Calculating the relevance of each block in the first block tree to the search request of the user based on the embedded model; Performing pruning operations on the blocks of the first block tree in order from small to large based on the correlation; The format of the first block of trees that has completed the pruning operation is converted to obtain the first search refined data.

6. The method according to claim 1, characterized in that The step of performing a second pruning operation on the second tree block based on the path generation model to obtain second search refined data includes: Calculating, based on the path generation model, a generation probability of each block in the second block tree under the search request condition of the current user; Performing pruning operations on the blocks of the second block tree in sequence based on the order of generation probability from small to large; The format of the second block tree that has completed the pruning operation is converted to obtain the second search refined data.

7. The method according to claim 6, characterized in that The calculating, based on the path generation model, the path generation probability of each block in the second block tree under the search request condition of the current user includes: Constructing a Token tree corresponding to the second block tree, wherein the Token tree comprises a plurality of sequences, and each sequence comprises at least one node; Calculate the first probability of each node in the same sequence of the token tree based on the path generation model; Each node of the token tree is traversed based on a depth-first algorithm, and for any traversed node, a generation probability of the current node is determined based on the path of the current node in the token tree and the first probability of each node of the token tree.

8. A large language model enhancement device based on Internet search, characterized in that: include: An acquisition module, used to obtain retrieval result data in response to a user's search request; A first processing module, configured to construct a first block tree based on the search result data, and perform a first pruning operation on the first block tree based on an embedded model to obtain first search refined data; A second processing module, configured to construct a second block tree based on the first search and refinement data, wherein the granularity of the second block tree is smaller than that of the first block tree; The second processing module is further used to perform a second pruning operation on the second block tree based on the path generation model to obtain second search refined data, and the second search refined data is used to input the large language model to generate an answer that matches the search request.

9. An electronic device, characterized in that: The electronic device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

Images