Search engine knowledge index generation method and related equipment
By segmenting the original text and generating annotations and contextual information, the problems of difficult information filtering and long computation time in traditional search engines are solved, resulting in more efficient and accurate search results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-31
- Publication Date
- 2026-03-13
AI Technical Summary
Traditional search engines rely primarily on keywords to find relevant web pages, requiring users to carefully filter or combine multiple web pages to obtain the information they need, and online computation has a long latency.
By segmenting the original text into multiple text blocks, adding annotation and contextual information, and generating sparse and dense indexes, the completeness and accuracy of the text block information are improved.
It improves the hit rate of search results, reduces the need for users to filter information, saves online computing latency, and enhances the accuracy of search results.
Smart Images
Figure CN121658573A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence (AI), and more particularly to a method for generating knowledge indexes for a search engine and related equipment. Background Technology
[0002] Since the 1990s, information retrieval technology, exemplified by "search engines," has advanced rapidly, significantly enhancing people's ability to retrieve information from massive amounts of internet data.
[0003] Search can generally be divided into two main parts: offline and online. In the offline stage, the system performs preprocessing operations on documents and creates an index. In the online stage, after receiving a user's search request, the system first parses the request and sends the parsed query content to the indexing system. It then assesses the relevance of documents using methods such as the classic BM25 probabilistic retrieval model or deep learning-based retrieval methods, selecting the top K most relevant documents. Subsequently, the system sorts these documents using a more sophisticated ranking algorithm before finally displaying them to the user. This index-centric "index-recall-ranking" retrieval framework is widely used in many information retrieval systems. However, current traditional search engines mainly rely on keywords to find relevant web pages. After receiving a user's query, they use the index to quickly locate web pages containing the keywords and display them to the user. Users often need to carefully filter or synthesize multiple web pages to obtain the information they need. Summary of the Invention
[0004] This application provides a method and related equipment for generating a search engine knowledge index. Using this application is beneficial to improving the accuracy of search results.
[0005] In a first aspect, embodiments of this application provide a method for generating a search engine knowledge index.
[0006] The index generation device acquires the original text; the index generation device segments the original text to obtain multiple text blocks; the index generation device generates text annotation information for each text block based on each text block in the multiple text blocks, and / or generates contextual information for each text block based on each text block, the contextual information of each text block including basic contextual information of each text block and identifiers of associated text blocks of each text block; the index generation device generates an index for each text block based on the annotation information and / or contextual information of each text block.
[0007] Within this framework, multiple text blocks can have partially identical, completely identical, or distinct granularities. These granularities include, but are not limited to, chapters, paragraphs, sentences, phrases, and words.
[0008] The contextual information of a text block includes at least one of the following:
[0009] The title of the text block, the creation time of the original text to which the text block belongs, the source site of the text block, the creator, the title and link of the parent text block, the title and link of the sibling text blocks, and the title, identifier and link of the inbound / outbound text blocks.
[0010] It can be seen that segmenting the original text into multiple text blocks improves the hit rate of search results, i.e., the accuracy of search results, and reduces the need for users to filter the information they need from the search results. It also saves online computation latency, making it suitable for large model proxy access scenarios. Secondly, adding annotation information to the text blocks obtained from the original text is equivalent to inferring deep information that the online model cannot understand before online inference, which helps the online model handle more complex user requests and improves the accuracy of search results. Furthermore, the inclusion of contextual information in the text block index enhances the completeness of information in individual text blocks, reducing inaccurate search results caused by incomplete index information, thereby improving the accuracy of search results.
[0011] In conjunction with the first aspect, in one possible implementation, the index generation device generates text annotation information for each text block based on each text block in a plurality of text blocks, including:
[0012] The index generation device determines the position and type of annotations in each text block based on each text block; the index generation device generates annotation information for each text block based on the position and type of annotations.
[0013] The types of annotations include, but are not limited to, questions, interpretations, and extensions.
[0014] It can be seen that by first determining the location and type of the annotation, and then generating annotation information based on the location and type of the annotation, the accuracy of the annotation information is improved.
[0015] In conjunction with the first aspect, in one possible implementation, the index generation device generates contextual information for each text block based on each text block, including:
[0016] The index generation device parses each text block to obtain the basic context information of each text block; the index generation device determines the associated topic corresponding to each text block based on each text block and the basic context information of each text block; the index generation device retrieves the associated text block of each text block from the text block library based on the associated topic of each text block and obtains the identifier of the associated text block.
[0017] It can be seen that determining the associated topics based on the basic contextual information of the text block and the text block itself helps to improve the accuracy of the determined associated topics; further, determining the associated text blocks based on the associated topics and adding the identifiers of the associated text blocks to the index during subsequent index construction helps to improve the breadth of information in the index.
[0018] In conjunction with the first aspect, in one possible implementation, the index generation device retrieves the associated text blocks for each text block from the text block library based on the associated topics corresponding to each text block, including:
[0019] The index generation device retrieves multiple reference text blocks from the text block library based on the associated topics corresponding to each text block. The reference text blocks are text blocks whose content includes the associated topics or whose titles include the associated topics. The index generation device calculates the degree of association between the multiple reference text blocks and each text block. The index generation device determines the text block with the highest degree of association among the multiple reference text blocks as the associated text block of each text block.
[0020] In conjunction with the first aspect, in one possible implementation, the index generation device retrieves the associated text blocks for each text block from the text block library based on the associated topics corresponding to each text block, including:
[0021] The index generation device matches the associated topics corresponding to each text block with the titles of text blocks in the text block library based on their relevance, and determines the text block with the highest relevance in the text block library as the associated text block for each text block.
[0022] In conjunction with the first aspect, in one possible implementation, the index generation device generates an index for each text block based on the annotation information of each text block and / or the contextual information of each text block, including:
[0023] The index generation device obtains multiple keywords from the annotation information and / or context information of each text block; the index generation device determines multiple word weights corresponding to the multiple keywords from the lexicon, and the sparse index of each text block is a sparse vector composed of the multiple word weights corresponding to the multiple keywords; the index generation device converts the annotation information and / or context information of each text block into a dense vector of fixed dimension, and the dense index of each text block is a dense vector of fixed dimension; wherein, the index of each text block includes the sparse index and the dense index of each text block.
[0024] It can be seen that generating an index for each text block based on the annotation information and / or the contextual information of each text block is beneficial to improving the depth and breadth of the index information, which in turn is beneficial to improving the accuracy of the retrieval results; using an index for text blocks composed of sparse indexes and dense indexes is beneficial to ensuring the uniqueness of the index, which in turn is beneficial to improving the accuracy of the retrieval results.
[0025] Secondly, embodiments of this application provide an index generation apparatus. The index generation apparatus includes an acquisition unit, a segmentation unit, and a generation unit.
[0026] The acquisition unit is used to acquire the raw text;
[0027] A segmentation unit is used to segment the original text into multiple text blocks;
[0028] The generation unit is configured to generate text annotation information for each text block based on each text block in a plurality of text blocks, and / or generate contextual information for each text block based on each text block, wherein the contextual information of each text block includes basic contextual information of each text block and identifiers of associated text blocks of each text block; the index generation device generates an index for each text block based on the annotation information and / or contextual information of each text block.
[0029] In conjunction with the second aspect, in one possible implementation, regarding the generation of text annotation information for each text block based on each text block in a plurality of text blocks, the generation unit is specifically used for:
[0030] The location and type of annotations in each text block are determined based on each text block; annotation information for each text block is generated based on the location and type of annotations.
[0031] In conjunction with the second aspect, in one possible implementation, in generating contextual information for each text block based on each text block, the generation unit is specifically used for:
[0032] Each text block is parsed to obtain its basic contextual information; based on each text block and its basic contextual information, the associated topic corresponding to each text block is determined; the index generation device retrieves the associated text blocks of each text block from the text block library based on the associated topic corresponding to each text block, and obtains the identifier of the associated text blocks.
[0033] In conjunction with the second aspect, in one possible implementation, regarding the aspect of retrieving the associated text blocks for each text block from the text block library based on the associated topics corresponding to each text block, the generation unit is specifically used for:
[0034] Based on the associated topics corresponding to each text block, multiple reference text blocks are obtained from the text block library. The reference text blocks are text blocks whose content includes the associated topics or whose titles include the associated topics. The degree of association between the multiple reference text blocks and each text block is calculated, and the text block with the highest degree of association among the multiple reference text blocks is determined as the associated text block of each text block.
[0035] In conjunction with the second aspect, in one possible implementation, regarding the aspect of retrieving the associated text blocks for each text block from the text block library based on the associated topics corresponding to each text block, the generation unit is specifically used for:
[0036] The associated topics corresponding to each text block are matched with the titles of text blocks in the text block library based on their relevance. The text block with the highest relevance in the text block library is then identified as the associated text block for each text block.
[0037] In conjunction with the second aspect, in one possible implementation, regarding the generation of an index for each text block based on the annotation information and / or contextual information of each text block, the generation unit is specifically used for:
[0038] Multiple keywords are obtained from the annotation information and / or context information of each text block; multiple word weights corresponding to the multiple keywords are determined from the thesaurus, and the sparse index of each text block is a sparse vector composed of the multiple word weights corresponding to the multiple keywords; the annotation information and / or context information of each text block are converted into a dense vector of fixed dimension, and the dense index of each text block is a dense vector of fixed dimension; wherein, the index of each text block includes the sparse index and the dense index of each text block.
[0039] Thirdly, embodiments of this application provide an index generation apparatus, including a processor and a memory. The memory is used to store program code. The processor is used to invoke the program code stored in the memory to execute the method provided in the first aspect or any possible implementation of the first aspect.
[0040] Fourthly, embodiments of this application provide a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform a method as provided in any possible implementation of the first aspect.
[0041] Fifthly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the method provided in any possible implementation of the first aspect.
[0042] It is understood that the index generation apparatus described in the second or third aspect above is used to execute any of the methods provided in the first aspect, and the computer storage medium described in the fourth aspect and the computer program product described in the fifth aspect are both used to implement any of the methods provided in the first aspect. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description
[0043] Figure 1 A schematic diagram of a system architecture provided for an embodiment of this application;
[0044] Figure 2 This application provides a schematic diagram of an index construction process.
[0045] Figure 3 A schematic diagram of a search process provided in an embodiment of this application;
[0046] Figure 4 A flowchart illustrating a search engine knowledge index generation method provided in this application embodiment;
[0047] Figure 5 This application provides a text segmentation diagram as an embodiment of the present application.
[0048] Figure 6 A schematic diagram illustrating the generation of annotation information for a text block, provided as an embodiment of this application;
[0049] Figure 7 A schematic diagram illustrating the generation of contextual information for a text block, provided as an embodiment of this application;
[0050] Figure 8 This is a schematic diagram of the structure of an index generation device provided in an embodiment of this application;
[0051] Figure 9 This is a schematic diagram of another index generation device provided in an embodiment of this application. Detailed Implementation
[0052] The terms “first,” “second,” “third,” and “fourth,” etc., used in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order.
[0053] "Multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating three possible relationships. For example, A and / or B means: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0054] The embodiments of this application will now be described with reference to the accompanying drawings.
[0055] See Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided for an embodiment of this application. Figure 1 As shown, the system architecture includes a first device 101 and multiple second devices 102. The first device 101 may be one or more.
[0056] The first device 101 can be a server, such as a cloud server, distributed server, rack server, blade server, tower server, etc. The second device 102 can be a server, terminal device, etc. The terminal device can be a smartphone, smartwatch, smart bracelet, desktop computer, laptop, tablet, etc.
[0057] Offline search process, such as Figure 2 As instructed, the first device 101 acquires raw data, which may be an original webpage, an original document, or data in other formats; the first device 101 performs basic parsing processing on the raw data to obtain raw text; the first device 101 constructs an index based on the raw text. The index includes sparse indexes and dense indexes. Multiple indexes can be obtained in this manner.
[0058] The first device 101 builds an index based on the original text, the specific process of which is as follows: Figure 2 As shown, the first device 101 segments the original text to obtain multiple text blocks; wherein some or all of the multiple text blocks have different granularities, or the granularities of the multiple text blocks are all the same. Optionally, before segmenting the original text, the original text is preprocessed to remove noise and useless information. The first device 101 generates annotation information for each text block based on each text block in the multiple text blocks, and / or generates contextual information for each text block based on each text block. The first device 101 generates an index for each text block based on the annotation information and / or contextual information of each text block. The first device 101 stores the obtained index of the text blocks.
[0059] Online search process, such as Figure 3As shown, the user sends a query request to the first device 101 through the second device 102. The first device 101 converts the query request into a query statement that is easy for the search engine to execute, such as extracting keywords from the query request as the query statement, or decomposing the original request into subtasks to obtain the query statement. The first device 101 performs a retrieval based on the query statement, matching the query statement with the indexes for relevance, so as to obtain one or more indexes with a relevance higher than the preset relevance. For multiple indexes, the multiple indexes are sorted by a ranking model, such as a logistic regression (LR) model or an XGBoost model. Then, the top K indexes are obtained, and the text blocks corresponding to the K indexes are obtained. The text blocks corresponding to the K indexes are used as the query results corresponding to the query statement. The answer is synthesized based on the query results to obtain the final reply corresponding to the user's request, and the final reply is sent to the second device 102 to provide feedback to the user.
[0060] As can be seen, in the index construction scheme of this embodiment, firstly, by segmenting the original text into multiple text blocks of different granularities, it is beneficial to improve the hit rate of search results and reduce the need for users to filter the required information from the search results; it also saves online computing latency and is beneficial for application in large model proxy access scenarios; secondly, by adding annotation information to the text blocks obtained based on the original text, it is equivalent to inferring the deep information that the online model cannot understand in advance before online inference, which is beneficial to assist the online model in handling more complex user requests; in the index of text blocks, the contextual information of the associated text blocks is included, which improves the completeness of the information of a single text block and reduces the occurrence of inaccurate query results due to incomplete index information of text blocks.
[0061] The index building process of this application is described in detail below.
[0062] See Figure 4 , Figure 4 This is a flowchart illustrating a method for generating a search engine knowledge index, provided as an embodiment of this application. Figure 4 As shown, the method includes:
[0063] S401, The index generation device obtains the original text.
[0064] In one example, the indexing device acquires data in other formats, such as original web pages, original documents, or other formats. The original web page can be a Hypertext Markup Language (HTML) web page. HTML is a standard markup language used to create web pages. It defines the structure and content of a web page through tags and attributes. The indexing device removes the tags from the HTML web page, extracting the usable text to obtain the original text corresponding to the original web page. For original documents, such as doc, pdf, or HTML documents, the indexing device converts non-HTML original documents to HTML documents. The indexing device then removes the tags from the HTML document, extracting the usable text to obtain the original text corresponding to the original document.
[0065] It should be noted that the index generation device is the first device 101, or a functional unit or functional module in the first device 101.
[0066] S402, The index generation device segments the original text to obtain multiple text blocks.
[0067] This involves multiple text blocks with the same granularity, or some or all of the text blocks having different granularities. This granularity includes, but is not limited to, chapters, paragraphs, sentences, phrases, and words. Of course, this granularity includes some or all of a chapter, paragraph, sentence, phrase, or word. In a specific example, such as... Figure 5 As shown in the upper part, the index generation device divides the original text into two text blocks, namely Chapter 1 and Chapter 2; the index generation device divides Chapter 1 into two text blocks, namely Paragraph 1 and Paragraph 2, and divides Chapter 2 into two text blocks, namely Paragraph 3 and Paragraph 4; similarly, the index generation device can further divide Paragraph 1, Paragraph 2, Paragraph 3, and Paragraph 4 into even smaller granular text blocks, such as sentences, phrases, and words. Thus, the granularity of the text blocks obtained by the index generation device from the original text includes: chapters, paragraphs, sentences, phrases, and words.
[0068] like Figure 5As shown in the lower half, the original text comprises three chapters. The index generation device segments the original text into three text blocks: Chapter 1, Chapter 2, and Chapter 3. Chapter 1 is segmented into two text blocks: Chapter 1-Paragraph 1 and Chapter 1-Paragraph. Chapter 2 is segmented into two text blocks: Chapter 2-Paragraph 1 and Chapter 2-Paragraph. Chapter 1-Paragraph 1 is segmented into three text blocks: Chapter 1-Paragraph 1-Sentence 1, Chapter 1-Paragraph 1-Sentence 2, and Chapter 1-Paragraph 1-Sentence 3. It should be noted that other chapters and paragraphs can also be further segmented in this manner, with the segmentation results shown in the figure. Figure 5 The lower half is not shown. Thus, the text blocks obtained by segmenting the original text include: Chapter 1, Chapter 2, Chapter 3, Chapter 1-Paragraph 1, Chapter 1-Paragraph 2, Chapter 2-Paragraph 1, Chapter 2-Paragraph 2, Chapter 1-Paragraph 1-Sentence 1, Chapter 1-Paragraph 1-Sentence 2, and Chapter 1-Paragraph 1-Sentence, etc.
[0069] In one feasible implementation, the index generation device segments the original text to obtain multiple text blocks through a text segmenter; alternatively, the text segmenter may be based on a neural network or other methods, which are not limited here.
[0070] Furthermore, when the text segmenter is implemented based on a neural network, it can be trained in the following way:
[0071] Obtain a first training sample, which includes a first text and multiple first text blocks, wherein the multiple first text blocks are obtained by manually segmenting the first text; input the first text into a first neural network model for processing to obtain multiple second text blocks; adjust the parameters in the first neural network model based on the multiple first text blocks, the multiple second text blocks and manually written rules to obtain a text segmenter.
[0072] It should be understood that the above training process can be iterated multiple times; the above training process can be implemented by an index generation device or other devices.
[0073] It's important to note that a text segmenter can be viewed as a tool that integrates manually defined rules and model recognition capabilities. The manually defined rules include segmenting sentences using periods and question marks, and segmenting paragraphs using carriage returns and tags. The model recognition capability includes segmenting text into chapters using a neural network model; in other words, determining which character in the text sequence marks the start of a new chapter. Of course, by training the neural network model, it's also possible to enable it to perform the functions corresponding to the manually defined rules.
[0074] S403, the index generation device generates text annotation information for each text block based on each text block in the multiple text blocks, and / or generates contextual information for each text block based on each text block.
[0075] In one feasible implementation, the index generation device determines the position and type of annotations within each of multiple text blocks, and generates annotation information for that text block based on each text block and the position and type of its annotations. It should be understood that a text block may contain one or more annotations.
[0076] The types of annotations include, but are not limited to, questions, interpretations, and extensions.
[0077] In one example, the index generation device determines the position and type of annotations in each of multiple text blocks using an annotation recognition model. Specifically, the index generation device inputs each text block into the annotation recognition model for processing to obtain the position and type of annotations in each text block. The index generation device then inputs the position and type of annotations in each text block, along with the text block itself, into the annotation generation model for processing to obtain the annotation information for each text block.
[0078] like Figure 6 As shown, the text block contains information related to Sichuan cuisine. The index generation device inputs the text block into the annotation recognition model for processing to obtain the location and type of the annotation. Figure 6 The triangle in the text block represents the position of the annotation; the index generation device inputs the position and type of the annotation, as well as the text block itself, into the annotation generation model for processing to obtain the annotation information of the text block. For example... Figure 6 As shown, the text block has 5 annotations. For example, the first annotation is a question type, located at "20", and its content is "Where did Sichuan cuisine originate?". The fourth annotation is a supplementary explanation type, located at "217", and its content is "Shuzhong is the Sichuan region". The second annotation is an extension type, located at "118", and its content is "Sichuan cuisine had already begun to emerge during the Warring States period".
[0079] In one example, the annotation recognition model was trained as follows:
[0080] Obtain a second training sample, which includes the position and type of the first batch of annotations of the second text block and the second text block. The position and type of the first batch of annotations are determined manually based on the second text block. Input the second text block into the second neural network model for processing to obtain the position and type of the second annotation of the second text block. Adjust the parameters in the second neural network model based on the position and type of the first batch of annotations and the position and type of the second annotation to obtain the annotation recognition model.
[0081] It should be understood that the above training process can be iterated multiple times; the above training process can be implemented by an index generation device or other devices.
[0082] In one example, the annotation generation model was trained as follows:
[0083] A third training sample is obtained, which includes a third text block and the first batch of annotation information of the third text block. The first batch of annotation information is obtained by manually annotating the third text block. The third text block is input into the third neural network model for processing to obtain the second annotation information of the third text block. The parameters in the third neural network model are adjusted based on the first batch of annotation information and the second batch of annotation information to obtain the annotation generation model.
[0084] It should be understood that the above training process can be iterated multiple times; the above training process can be implemented by an index generation device or other devices.
[0085] In one feasible implementation, the index generation device parses each text block to obtain the basic context information of each text block; determines the associated topic corresponding to each text block based on each text block and its basic context information; and retrieves the associated text blocks of each text block from the text block library based on the associated topics of each text block and obtains the identifier of the associated text blocks. The basic context information of each text block includes the basic context information of each text block and the identifier of the associated text blocks of each text block.
[0086] It should be noted that determining the associated topic for each text block based on its underlying contextual information can be achieved using a topic association model, which can be implemented in the following way:
[0087] A fourth training sample is obtained, which includes a fourth text block, basic context information of the fourth text block, and a first associated topic corresponding to the fourth text block. The first associated topic corresponding to the fourth text block is manually set. The fourth text block and its basic context information are input into a fourth neural network model for processing to obtain a second associated topic of the fourth text block. The parameters in the fourth neural network model are adjusted based on the first and second associated topics to obtain a topic association model.
[0088] It should be understood that the above training process can be iterated multiple times; the above training process can be implemented by an index generation device or other devices.
[0089] In one possible implementation, the index generation device retrieves the associated text blocks for each text block from the text block library based on the associated topics corresponding to each text block, including:
[0090] The index generation device retrieves multiple reference text blocks from the text block library based on the associated topics corresponding to each text block. The reference text blocks are text blocks whose content includes the associated topics or whose titles include the associated topics. The index generation device calculates the degree of association between the multiple reference text blocks and each text block, and determines the text block with the highest degree of association among the reference text blocks as the associated text block of each text block.
[0091] In another feasible implementation, the index generation device retrieves the associated text blocks for each text block from the text block library based on the associated topics corresponding to each text block, including:
[0092] The index generation device matches the associated topics corresponding to each text block with the titles of text blocks in the text block library based on their relevance. It identifies text blocks in the text block library with a relevance higher than the relevance threshold as associated text blocks for each text block, or identifies the text block with the highest relevance in the text block library as associated text blocks for each text block.
[0093] It should be noted that there can be one or more associative text blocks.
[0094] In one example, the contextual information of each text block includes at least one of the following: the title of the text block, the creation time of the original text to which the text block belongs, the source site of the text block, the creator, the title and link of the parent text block, the title and link of the sibling text blocks, and the titles, identifiers, and links of the inbound / outbound link text blocks. It should be noted that the contextual information of a text block is not limited to the parts disclosed above, and may include other information, which is not limited here.
[0095] Here, the parent text block refers to the text block at the next higher level, and the sibling text block refers to text blocks of the same granularity obtained by segmenting the same text block, such as... Figure 5As shown, Chapter 1-Paragraph 1 and Chapter 1-Paragraph 2 are sibling text blocks, and Chapter 1 is the parent text block of both Chapter 1-Paragraph 1 and Chapter 1-Paragraph 2.
[0096] Inbound / outbound text blocks refer to text blocks that contain hyperlinks, or text blocks that are hyperlinked to other text blocks. For example, if a text block contains "City A" and hyperlinks to the encyclopedia page for City A, then the encyclopedia page is an inbound text block containing "City A," and the text block containing "City A" is an outbound text block for that encyclopedia page.
[0097] like Figure 7 As shown, the index generation device for, Figure 7 The text block shown in the upper part is parsed to obtain its basic context information, which includes the text block's title "WHO: Summer Respiratory Virus Infection Rates Surge, Possibility of Resurgence Cannot Be Ruled Out," the original text creation time "2024 / 08 / 06," the parent text block title, the sibling text block titles, the text block's subtitles, and the outgoing text blocks. The index generation device determines the associated topic "Treatment Methods for Respiratory Virus Infections" based on the text block's basic context information and the text block itself. The index generation device then retrieves the associated text blocks from the text block database based on the associated topic and obtains the identifiers of the associated text blocks.
[0098] S404. The index generation device generates an index for each text block based on the annotation information and / or contextual information of each text block.
[0099] The index of each text block includes a sparse index and a dense index. The index generation device obtains multiple keywords from the annotation information and / or context information of each text block, determines multiple word weights corresponding to the multiple keywords from the thesaurus, and the sparse index of each text block is a sparse vector composed of multiple word weights corresponding to the multiple keywords. The index generation device converts the annotation information and / or context information of each text block into a dense vector of fixed dimension, and the dense index of each text block is the dense vector of fixed dimension.
[0100] It should be understood that the annotation information and / or contextual information of each text block can be regarded as a text, and the index generation device digitizes the text to obtain a dense vector of fixed dimensions.
[0101] As can be seen, in this embodiment, firstly, by segmenting the original text into multiple text blocks of different granularities, the hit rate of search results is improved, and the need for users to filter the required information from the search results is reduced; it also saves online computation latency, which is beneficial for application in large model proxy access scenarios; secondly, adding annotation information to the text blocks obtained based on the original text is equivalent to inferring deep information that is difficult for the online model to understand in advance before online inference, which helps the online model to handle more complex user requests; in the index of the text blocks, the contextual information of the associated text blocks is included, which improves the completeness of the information of a single text block and reduces the occurrence of inaccurate query results due to incomplete index information of the text blocks, thereby improving search efficiency and the accuracy of search results.
[0102] Reference Figure 8 The diagram shown is a structural schematic of an index generation device provided in an embodiment of this application. Figure 8 As shown, the index generation device 800 includes:
[0103] Acquisition unit 801 is used to acquire the original text;
[0104] The segmentation unit 802 is used to segment the original text to obtain multiple text blocks;
[0105] The generation unit 803 is configured to generate text annotation information for each text block based on each text block in a plurality of text blocks, and / or generate contextual information for each text block based on each text block, wherein the contextual information for each text block includes basic contextual information for each text block and identifiers of associated text blocks for each text block; the index generation device generates an index for each text block based on the annotation information and / or contextual information for each text block.
[0106] In one possible implementation, in generating text annotation information for each text block based on each text block in a plurality of text blocks, the generation unit 803 is specifically used for:
[0107] The location and type of annotations in each text block are determined based on each text block; annotation information for each text block is generated based on the location and type of annotations.
[0108] In one possible implementation, in generating contextual information for each text block based on each text block, the generation unit 803 is specifically used for:
[0109] Each text block is parsed to obtain its basic contextual information; based on each text block and its basic contextual information, the associated topic corresponding to each text block is determined; the index generation device retrieves the associated text blocks of each text block from the text block library based on the associated topic corresponding to each text block, and obtains the identifier of the associated text blocks.
[0110] In one possible implementation, in retrieving the associated text blocks of each text block from the text block library based on the associated topics corresponding to each text block, the generation unit 803 is specifically used for:
[0111] Based on the associated topics corresponding to each text block, multiple reference text blocks are obtained from the text block library. The reference text blocks are text blocks whose content includes the associated topics or whose titles include the associated topics. The degree of association between the multiple reference text blocks and each text block is calculated, and the text block with the highest degree of association among the multiple reference text blocks is determined as the associated text block of each text block.
[0112] In one possible implementation, in retrieving the associated text blocks of each text block from the text block library based on the associated topics corresponding to each text block, the generation unit 803 is specifically used for:
[0113] The associated topics corresponding to each text block are matched with the titles of text blocks in the text block library based on their relevance. The text block with the highest relevance in the text block library is then identified as the associated text block for each text block.
[0114] In one possible implementation, in generating an index for each text block based on the annotation information and / or contextual information of each text block, the generation unit 803 is specifically used for:
[0115] Multiple keywords are obtained from the annotation information and / or context information of each text block; multiple word weights corresponding to the multiple keywords are determined from the thesaurus, and the sparse index of each text block is a sparse vector composed of the multiple word weights corresponding to the multiple keywords; the annotation information and / or context information of each text block are converted into a dense vector of fixed dimension, and the dense index of each text block is a dense vector of fixed dimension; wherein, the index of each text block includes the sparse index and the dense index of each text block.
[0116] It is worth noting that the specific functional implementation of the index generation device 800 is described in the relevant description of the search engine knowledge index generation method above. For example, the acquisition unit 801 is used to execute the relevant content of S401, the segmentation unit 802 is used to execute the relevant content of S402, and the generation unit 803 is used to execute the relevant content of S403 and S404. Each unit or module in the index generation device 800 can be individually or entirely merged into one or more other units or modules, or some of the units or modules can be further divided into multiple functionally smaller units or modules. This achieves the same operation without affecting the technical effect of the embodiments of this application. The above-mentioned units or modules are based on logical function division. In practical applications, the function of one unit (or module) is implemented by multiple units (or modules), or the function of multiple units (or modules) is implemented by one unit (or module).
[0117] Based on the description of the above method embodiments and related device embodiments, please refer to... Figure 9 This application also provides a schematic diagram of the structure of another index generation device 900. Figure 9 The index generation device 900 shown includes a memory 901, a processor 902, a communication interface 903, and a bus 904. The memory 901, processor 902, and communication interface 903 are interconnected via the bus 904.
[0118] Optionally, the memory 901 can be ROM, static storage device, dynamic storage device, or RAM.
[0119] Memory 901 can store programs. When the program stored in memory 901 is executed by processor 902, processor 902 and communication interface 903 are used for execution. Figure 2 The steps of the search engine knowledge index generation method in the illustrated embodiment.
[0120] The processor 902 employs a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), GPU, or one or more integrated circuits to execute relevant programs to achieve... Figure 4 The illustrated embodiment is a method for generating a search engine knowledge index.
[0121] The processor 902 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the traffic scheduling in this application can be completed through integrated logic circuits in the hardware of the processor 902 or instructions in software form. Optionally, the processor 902 can be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor is a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. Optional software modules are located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 901. The processor 902 reads the information in memory 901 and, in conjunction with its hardware, completes the functions required by the units included in the index generation apparatus 800 of the embodiments of this application, or executes the search engine knowledge index generation method of the method embodiments of this application.
[0122] The communication interface 903 uses transceiver-related devices, such as, but not limited to, transceivers, to enable communication between the index generation device 900 and other devices or communication networks.
[0123] Bus 904 may include a pathway for transmitting information between various components of index generation apparatus 900 (e.g., memory 901, processor 902, communication interface 903).
[0124] It should be noted that, although Figure 9 The index generation apparatus 900 shown only illustrates the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, the index generation apparatus 900 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the index generation apparatus 900 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the index generation apparatus 900 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 9 All the devices shown.
[0125] This application also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface to implement the search engine knowledge index generation method of this application.
[0126] Optionally, as one implementation, the chip further includes a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the search engine knowledge index generation method.
[0127] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps of any of the above methods.
[0128] This application also provides a computer program product containing instructions. When the computer program product is run on a computer or processor, it causes the computer or processor to perform one or more steps of any of the methods described above.
[0129] Those skilled in the art will appreciate that the functionality described in conjunction with the various illustrative logic blocks, modules, and algorithmic steps disclosed herein can be implemented by hardware, software, firmware, or any combination thereof. If implemented in software, the functionality described by the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may comprise a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium that includes any medium facilitating the transfer of a computer program from one place to another (e.g., based on a communication protocol). In this way, the computer-readable medium may substantially correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. A computer program product may comprise a computer-readable medium.
[0130] By way of example and not limitation, such computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection is properly referred to as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but are specifically referring to non-temporary tangible storage media. As used herein, disks and optical discs include Compact Discs (CDs), Laser Discs, Optical Discs, Digital Versatile Discs (DVDs), and Blu-ray Discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.
[0131] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein. Furthermore, in some aspects, the functions described in the various illustrative logic blocks, modules, and steps described herein are provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, the techniques can be fully implemented within one or more circuit or logic elements.
[0132] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the division of units is merely a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Optionally, the coupling, direct coupling, or communication connection shown or discussed between them may be through some interfaces, indirect coupling or communication connection of devices or units, such as electrical, mechanical, or other forms.
[0133] Optionally, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0134] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of this application are generated.
[0135] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.
Claims
1. A method for generating a knowledge index for a search engine, characterized in that, The method includes: Get the original text; The original text is segmented to obtain multiple text blocks; The process involves generating text annotation information for each text block based on each of the plurality of text blocks, and / or generating contextual information for each text block based on each text block, wherein the contextual information for each text block includes basic contextual information for each text block and identifiers of associated text blocks for each text block. An index for each text block is generated based on the annotation information and / or the contextual information of each text block.
2. The method according to claim 1, characterized in that, The step of generating text annotation information for each text block based on each of the plurality of text blocks includes: The location and type of the annotation in each text block are determined based on each text block. Annotation information for each text block is generated based on each text block, the location of the annotation, and the type of the annotation.
3. The method according to claim 1 or 2, characterized in that, The step of generating contextual information for each text block based on each text block includes: Each text block is parsed to obtain the basic contextual information of each text block; Based on each text block and its underlying contextual information, determine the associated topic corresponding to each text block; Based on the associated topics corresponding to each text block, the associated text blocks of each text block are retrieved from the text block library, and the identifier of the associated text blocks is obtained.
4. The method according to claim 3, characterized in that, The step of retrieving associated text blocks for each text block from the text block library based on the associated topics corresponding to each text block includes: Based on the associated topics corresponding to each text block, multiple reference text blocks are obtained from the text block library. The reference text blocks are text blocks whose content includes the associated topics, or text blocks whose titles include the associated topics. Calculate the degree of association between the plurality of reference text blocks and each text block. The text block with the highest degree of association among the plurality of reference text blocks is determined as the associated text block of each text block.
5. The method according to claim 3, characterized in that, The step of retrieving associated text blocks for each text block from the text block library based on the associated topics corresponding to each text block includes: The associated topics corresponding to each text block are matched with the titles of text blocks in the text block library for relevance, and the text block with the highest relevance in the text block library is determined as the associated text block of each text block.
6. The method according to any one of claims 1-4, characterized in that, The step of generating an index for each text block based on the annotation information and / or contextual information of each text block includes: Multiple keywords are obtained from the annotation information and / or contextual information of each text block; The multiple word weights corresponding to the multiple keywords are determined from the lexicon, and the sparse index of each text block is a sparse vector composed of the multiple word weights corresponding to the multiple keywords. The annotation information and / or contextual information of each text block are converted into a dense vector of fixed dimension, and the dense index of each text block is the dense vector of fixed dimension; The index of each text block includes a sparse index and a dense index for each text block.
7. An index generation apparatus, characterized in that, The device includes: The acquisition unit is used to acquire the raw text; A segmentation unit is used to segment the original text to obtain multiple text blocks; A generation unit is configured to generate text annotation information for each text block based on each of the plurality of text blocks; the index information for each text block includes, and / or, the context information for each text block is generated based on each text block, the context information for each text block including basic context information for each text block and identifiers of associated text blocks for each text block; and to generate an index for each text block based on the annotation information and / or the context information for each text block.
8. The apparatus according to claim 7, characterized in that, In the aspect of generating text annotation information for each of the plurality of text blocks, the generation unit is specifically used for: The location and type of the annotation in each text block are determined based on each text block. Annotation information for each text block is generated based on each text block, the location of the annotation, and the type of the annotation.
9. The apparatus according to claim 7 or 8, characterized in that, In the aspect of generating contextual information for each text block based on each text block, the generation unit is specifically configured to: Each text block is parsed to obtain the basic contextual information of each text block; Based on each text block and its underlying contextual information, determine the associated topic corresponding to each text block; Based on the associated topics corresponding to each text block, the associated text blocks of each text block are retrieved from the text block library, and the identifier of the associated text blocks is obtained.
10. The apparatus according to claim 9, characterized in that, In the aspect of retrieving associated text blocks for each text block from the text block library based on the associated topics corresponding to each text block, the generation unit is specifically used for: Based on the associated topics corresponding to each text block, multiple reference text blocks are obtained from the text block library. The reference text blocks are text blocks whose content includes the associated topics, or text blocks whose titles include the associated topics. Calculate the degree of association between the plurality of reference text blocks and each text block. The text block with the highest degree of association among the plurality of reference text blocks is determined as the associated text block of each text block.
11. The apparatus according to claim 9, characterized in that, In the aspect of retrieving associated text blocks for each text block from the text block library based on the associated topics corresponding to each text block, the generation unit is specifically used for: The associated topics corresponding to each text block are matched with the titles of text blocks in the text block library for relevance, and the text block with the highest relevance in the text block library is determined as the associated text block of each text block.
12. The apparatus according to any one of claims 7-11, characterized in that, In the aspect of generating an index for each text block based on the annotation information and / or the contextual information of each text block, the generation unit is specifically used for: Multiple keywords are obtained from the annotation information and / or contextual information of each text block; The multiple word weights corresponding to the multiple keywords are determined from the lexicon, and the sparse index of each text block is a sparse vector composed of the multiple word weights corresponding to the multiple keywords. The annotation information and / or contextual information of each text block are converted into a dense vector of fixed dimension, and the dense index of each text block is the dense vector of fixed dimension; The index of each text block includes a sparse index and a dense index for each text block.
13. An index generation apparatus, characterized in that, The method includes a processor and a memory, wherein the memory is used to store program code, and the processor is used to execute the program code to implement the method according to any one of claims 1 to 6.
14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.
15. A computer program product that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-6.