A text retrieval method, device, equipment, medium and product

Abstract

The application provides a text retrieval method, device, equipment, medium and product, and relates to the technical field of artificial intelligence. The method comprises the following steps: analyzing a text to generate semantic objects with type identifiers; inputting the text content of each object into a pre-trained text embedding model to obtain a content vector, and obtaining a corresponding type embedding vector from a type embedding query table according to the type identifier; splicing the two vectors to obtain a joint vector; constructing a vector index based on all joint vectors; receiving a user query instruction; and searching in the vector index according to the query instruction to obtain a search result. Thus, the resource utilization rate can be improved during text retrieval, the accuracy of text retrieval can be improved, and the user experience can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a text retrieval method, apparatus, device, medium, and product. Background Technology

[0002] With the popularization of artificial intelligence technology, semantic-based vector retrieval has become an important way to process unstructured data. Vector databases (VDs) convert text into vectors through embedding models and achieve efficient retrieval by leveraging similarity search. They are now widely used for finding text, images, and other content.

[0003] To further improve accuracy, the industry often combines traditional keyword-based retrieval (such as BM25) with vector retrieval to form a hybrid search (HS) system. For example, existing solutions divide text into multiple paragraphs, store them in two databases respectively, query in parallel during retrieval and merge the results, and then determine the final content through deduplication and reordering.

[0004] However, this method still has obvious limitations when dealing with texts with clear structures, such as academic papers: on the one hand, simply dividing the text into paragraphs does not utilize the paper's own structure (such as title, abstract, and body text), resulting in too many data blocks and increasing the storage and computational burden; on the other hand, the fusion retrieval strategy is relatively simple and fails to dynamically adjust the retrieval weights according to the query intent, affecting the relevance and accuracy of the results.

[0005] Therefore, existing text retrieval methods have low efficiency and accuracy when processing structured long texts. Summary of the Invention

[0006] This application provides a text retrieval method, apparatus, device, medium, and product to address the technical problem that existing text retrieval methods suffer from low efficiency and accuracy when processing structured long texts.

[0007] To solve the above-mentioned technical problems, this application is implemented as follows:

[0008] In a first aspect, embodiments of this application provide a text retrieval method, the method comprising: acquiring original text; parsing the text to generate multiple semantic objects carrying type identifiers, wherein the type identifiers include at least one of the following: title, abstract, introduction, body text, and conclusion; for each semantic object, inputting the text content of the semantic object into a pre-trained text embedding model to obtain a text content vector output by the text embedding model; searching for a corresponding type embedding vector from a pre-trained type embedding lookup table according to the type identifier of the semantic object; concatenating the text content vector with the type embedding vector to obtain a joint vector; constructing a vector index based on the multiple joint vectors of the multiple semantic objects; receiving a user's query instruction; performing a search in the vector index according to the query instruction to obtain search results.

[0009] Optionally, the retrieval process according to the query instruction in the vector index to obtain retrieval results includes: identifying the intent of the query instruction to obtain the query intent and the query entity set; performing vector retrieval in the vector index according to the query intent and the query entity set to obtain a set of candidate semantic objects and a vector cosine similarity score of each candidate semantic object in the set of candidate semantic objects; determining weight coefficients from a preset weight configuration table based on the query intent, processing the vector cosine similarity scores using the weight coefficients to obtain the final relevance score of each candidate semantic object; sorting the candidate semantic objects according to the final relevance score to obtain a sorting result, and returning the retrieval results based on the sorting result.

[0010] Optionally, when constructing a vector index based on multiple joint vectors of the multiple semantic objects, the method further includes: constructing an inverted index based on the multiple semantic objects; and performing vector retrieval in the vector index according to the query intent and the query entity set to obtain a set of candidate semantic objects and a vector cosine similarity score of each candidate semantic object in the set of candidate semantic objects, the method further includes: performing keyword retrieval in the inverted index according to the query intent and the query entity set to obtain a keyword relevance score of each candidate semantic object in the set of candidate semantic objects; and weighting the vector cosine similarity scores using the weight coefficients to obtain a final relevance score for each candidate semantic object, including: weighting and fusing the keyword relevance scores and the vector cosine similarity scores of each candidate semantic object using the weight coefficients to obtain the final relevance score for each candidate semantic object.

[0011] Optionally, constructing an inverted index based on the plurality of semantic objects includes: obtaining a preset type weight boosting coefficient corresponding to the type identifier of each semantic object among the plurality of semantic objects; performing word segmentation on the text content of the semantic objects to obtain multiple terms; constructing the inverted index based on the multiple terms, and associating and storing the type weight boosting coefficient with the multiple terms in the inverted index.

[0012] Optionally, the vector index is an HNSW graph index, which is constructed based on multiple joint vectors of the multiple semantic objects. This includes: for each joint vector, determining a candidate nearest neighbor set; selecting a nearest neighbor vector from the candidate nearest neighbor set; establishing a connection between the joint vector and the nearest neighbor vector; and constructing the HNSW graph index based on the connection between each joint vector and its corresponding nearest neighbor vector. Specifically, when selecting a nearest neighbor vector from the candidate nearest neighbor set, a preset composite distance calculation function is used to calculate the composite distance between the joint vector and each candidate nearest neighbor vector in the candidate nearest neighbor set. The nearest neighbor vector is selected from the candidate nearest neighbor set based on the composite distance. The composite distance calculation function is a weighted combination of the cosine distance between vectors and a type matching function. The type matching function takes a preset type guidance strength coefficient when the types of the joint vector and the candidate nearest neighbor vector are the same, and takes zero when the types are different.

[0013] Optionally, the query intent includes intent category and confidence level, and the query entity set includes academic entities identified from the user query command and the entity type of each academic entity; performing intent recognition on the query command to obtain the query intent and query entity set includes: preprocessing the query command to obtain preprocessed query text; wherein, the preprocessing includes at least one of the following: word segmentation, stop word filtering, and spelling correction; inputting the query text into a pre-trained academic query intent classifier to obtain the probability distribution output by the academic query intent classifier, taking the category with the highest probability in the probability distribution as the intent category, and taking the probability value corresponding to the category with the highest probability as the confidence level, wherein, the intent category includes at least one of the following: keyword retrieval, semantic retrieval, entity search, and review query; inputting the query text into a pre-trained academic named entity recognition model to obtain the academic entities in the query text output by the academic named entity recognition model and the entity type of each academic entity; constructing the query entity set based on the academic entities and the entity types, wherein, the academic entities include at least one of the following: scholar name, text title, and professional terminology.

[0014] Optionally, before performing vector retrieval in the vector index based on the query intent and the query entity set, the method further includes: filtering the vector index based on the query intent and the query entity set to obtain a candidate vector set; performing vector retrieval in the vector index based on the query intent and the query entity set to obtain a candidate semantic object set and a vector cosine similarity score of each candidate semantic object in the candidate semantic object set, including: processing the query instruction based on the query intent and the query entity set to obtain a query vector; performing vector retrieval in the candidate vector set based on the query vector to obtain a preset number of joint vectors sorted from high to low similarity with the query vector; determining the candidate semantic object set based on the semantic objects corresponding to each of the preset number of joint vectors, and determining the vector cosine similarity score of each candidate semantic object in the candidate semantic object set.

[0015] Optionally, filtering the vector index based on the query intent and the query entity set to obtain a candidate vector set includes: combining the entity types of each academic entity into an entity type combination; using the intent category and the entity type combination as an index, searching for the corresponding retrieval strategy descriptor in a predefined retrieval strategy knowledge base, wherein the retrieval strategy descriptor includes a semantic object type list and a metadata filtering condition template; according to the variable placeholders in the metadata filtering condition template, searching for the corresponding entity values ​​from the query entity set, replacing the variable placeholders with the entity values, and generating metadata filtering conditions; determining the target type range according to the semantic object type list, applying the metadata filtering conditions and the target type range to the vector index, and obtaining a joint vector that conforms to the metadata filtering conditions and the target type range, as the candidate vector set.

[0016] Optionally, keyword retrieval is performed in the inverted index according to the query intent and the query entity set to obtain the keyword relevance score of each candidate semantic object in the candidate semantic object set. This includes: generating a Boolean query tree according to the query intent and the query entity set, submitting the Boolean query tree to the inverted index, performing keyword retrieval to obtain the candidate semantic object set; and calculating the keyword relevance score for each candidate semantic object in the candidate semantic object set using a preset BM25 variant algorithm. This includes: obtaining the type weight boosting coefficient corresponding to the candidate semantic object from the inverted index for each candidate semantic object in the candidate semantic object set, and obtaining the term frequency of each term in the candidate semantic object; weighting the term frequency based on the type weight boosting coefficient to obtain the effective term frequency, and calculating the keyword relevance score of the candidate semantic object based on the effective term frequency.

[0017] Optionally, the query intent includes an intent category and a confidence level. Determining a weight coefficient from a preset weight configuration table based on the query intent includes: determining a high-confidence baseline weight vector and a low-confidence baseline weight vector corresponding to the intent category from the weight configuration table; if the confidence level is greater than or equal to a preset high-confidence threshold, then the high-confidence baseline weight vector is used as the weight coefficient; if the confidence level is less than a preset low-confidence threshold, then the low-confidence baseline weight vector is used as the weight coefficient; if the confidence level is between the low-confidence threshold and the high-confidence threshold, then the weight coefficient is calculated using linear interpolation based on the high-confidence baseline weight vector, the low-confidence baseline weight vector, the confidence level, the high-confidence threshold, and the low-confidence threshold; wherein the weight coefficient includes keyword weight, semantic weight, and metadata weight, and the sum of the keyword weight, the semantic weight, and the metadata weight is 1.

[0018] Optionally, the vector cosine similarity score is processed using the weight coefficients to obtain the final relevance score of each candidate semantic object, including: for each candidate semantic object in the candidate semantic object set, obtaining a preset metadata attribute set and the attribute weight corresponding to each metadata attribute; for each metadata attribute in the metadata attribute set, obtaining the attribute value corresponding to the metadata attribute in the metadata of the candidate semantic object, and obtaining the entity value corresponding to the metadata attribute in the query entity set, and calculating the attribute matching value between the attribute value and the entity value; weighting and summing each attribute matching value with the corresponding attribute weight to obtain the metadata matching score of the candidate semantic object; and weighting and summing the keyword relevance score, the vector cosine similarity score, and the metadata matching score of the candidate semantic object according to the keyword weight, the semantic weight, and the metadata weight in the weight coefficients to obtain the final relevance score of the candidate semantic object.

[0019] Secondly, embodiments of this application provide a text retrieval device, the device comprising: an acquisition module, configured to acquire original text, parse the text, and generate multiple semantic objects carrying type identifiers, wherein the type identifiers include at least one of the following: title, abstract, introduction, body text, and conclusion; an execution module, configured to, for each semantic object, input the text content of the semantic object into a pre-trained text embedding model to obtain a text content vector output by the text embedding model, search for a corresponding type embedding vector from a pre-trained type embedding lookup table according to the type identifier of the semantic object, concatenate the text content vector with the type embedding vector to obtain a joint vector, and construct a vector index based on the multiple joint vectors of the multiple semantic objects; the execution module is further configured to receive a user's query instruction, perform a search in the vector index according to the query instruction, and obtain search results.

[0020] Thirdly, embodiments of this application provide a network device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of a text retrieval method as described in the first aspect.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a text retrieval method as described in the first aspect.

[0022] Fifthly, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the steps of a text retrieval method as described in the first aspect.

[0023] In this embodiment, the text is first structured and parsed into semantic objects with type identifiers, thereby achieving fine-grained organization of the text content, facilitating retrieval for different chapters and improving processing efficiency. Furthermore, by fusing the text content vector and the type embedding vector into a joint vector and establishing an index, not only is semantic information preserved, but the structural features of the text are also incorporated, enhancing the expressive power of the index and thus improving the matching accuracy of the retrieval. During the retrieval stage, the retrieval results can be obtained by searching the previously constructed vector index according to the query instructions.

[0024] In summary, the optimization of the entire text retrieval process not only improves resource utilization but also significantly enhances retrieval efficiency and accuracy, thereby improving the user experience. Attached Figure Description

[0025] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0026] Figure 1 A flowchart illustrating a text retrieval method provided in an embodiment of this application.

[0027] Figure 2A A flowchart illustrating a text retrieval method provided in an embodiment of this application.

[0028] Figure 2B A flowchart illustrating a text retrieval method provided in an embodiment of this application.

[0029] Figure 3 This is a schematic diagram of the architecture of a text retrieval system provided in an embodiment of this application.

[0030] Figure 4 This is a schematic diagram of the components of a text retrieval system provided in an embodiment of this application.

[0031] Figure 5 This is a structural block diagram of a text retrieval device provided in an embodiment of this application.

[0032] Figure 6 This is a structural block diagram of a network device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] Figure 1 This application illustrates a text retrieval method according to an embodiment of the present application, such as... Figure 1 As shown, the method includes:

[0035] Step S101: Obtain the original text, parse the text, and generate multiple semantic objects carrying type identifiers;

[0036] The type identifier includes at least one of the following: title, abstract, introduction, main body, and conclusion;

[0037] Step S102: For each semantic object, input the text content of the semantic object into the pre-trained text embedding model to obtain the text content vector output by the text embedding model. According to the type identifier of the semantic object, find the corresponding type embedding vector from the pre-trained type embedding lookup table. Concatenate the text content vector and the type embedding vector to obtain a joint vector. Construct a vector index based on multiple joint vectors of multiple semantic objects.

[0038] Step S103: Receive the user's query instruction, perform a search in the vector index according to the query instruction, and obtain the search results.

[0039] It should be noted that, Figure 1 The method described first parses the original text (such as academic paper text). Its focus is on dividing the text into semantic objects with clear type identifiers, such as titles, abstracts, and body text, based on the structural characteristics of academic papers, rather than performing indiscriminate paragraph segmentation. For each semantic object, a pre-trained model obtains its text content vector, which is then combined with the corresponding type embedding vector obtained from a lookup table. These two vectors are concatenated to form a unified joint vector, which in turn constructs an overall vector index. The system then receives user query commands and performs searches within the vector index based on those commands to obtain the search results. By deeply integrating the structured information of the text into the vector representation and retrieval ranking process, the retrieval system can consider both content semantics and document structure, dynamically responding to user intent. This results in more accurate and intelligent retrieval, thereby improving retrieval efficiency and accuracy.

[0040] In one possible implementation, the system receives a user's query instruction, performs a search in the vector index based on the query instruction, and obtains the search results by: performing a vector search in the vector index based on the query intent and the query entity set to obtain a set of candidate semantic objects and a vector cosine similarity score for each candidate semantic object in the set; determining weight coefficients from a preset weight configuration table based on the query intent, processing the vector cosine similarity scores using the weight coefficients to obtain the final relevance score for each candidate semantic object; sorting the candidate semantic objects according to the final relevance score to obtain a ranking result, and returning the search results based on the ranking result.

[0041] It should be noted that when a user initiates a query, the system performs intent recognition on the query command to parse out the user's query intent and the set of query entities. Using this information, the system performs vector retrieval in the vector index to obtain a preliminary set of candidate semantic objects and their vector cosine similarity scores. Subsequently, based on the identified query intent, the system dynamically determines weight coefficients from a preset weight configuration table and uses these coefficients to process the preliminary similarity scores, thereby calculating a final relevance score that better reflects the user's true needs, and then sorts and returns the results accordingly.

[0042] Therefore, by identifying user query intent and combining it with entity sets for vector retrieval, the system can better meet users' actual needs. Simultaneously, by dynamically adjusting ranking weights based on query intent and optimizing the relevance scoring mechanism, the final ranking results better align with the user's query objectives. Finally, the ranking results are returned based on the weighted relevance scores, ensuring that highly relevant content is presented first, further enhancing the user experience.

[0043] In one possible implementation, when constructing a vector index based on multiple joint vectors of multiple semantic objects, the method further includes: constructing an inverted index based on multiple semantic objects;

[0044] In the case that vector retrieval is performed in the vector index based on the query intent and query entity set to obtain the candidate semantic object set and the vector cosine similarity score of each candidate semantic object in the candidate semantic object set, the method further includes: performing keyword retrieval in the inverted index based on the query intent and query entity set to obtain the keyword relevance score of each candidate semantic object in the candidate semantic object set.

[0045] The final relevance score of each candidate semantic object is obtained by weighting the vector cosine similarity score using weight coefficients. This includes weighting and fusing the keyword relevance score and vector cosine similarity score of each candidate semantic object using weight coefficients.

[0046] It should be noted that while constructing a vector index based on joint vectors, a traditional inverted index can also be constructed based on the same set of semantic objects. During the retrieval execution phase, the system simultaneously performs two search modes: on the one hand, it performs vector retrieval in the vector index to obtain vector cosine similarity scores; on the other hand, it performs keyword retrieval in the inverted index to obtain keyword relevance scores. In the subsequent fusion phase, weight coefficients determined according to the query intent from the weight configuration table are used to weight and fuse the two scores to generate the final ranking criteria.

[0047] This design enables the system to effectively combine the advantages of semantic similarity matching and traditional keyword matching, and balance their contributions in different query scenarios through an adjustable weight mechanism, thereby improving the overall relevance of search results and the system's ability to adapt to diverse queries.

[0048] In one possible implementation, an inverted index is constructed based on multiple semantic objects, including:

[0049] For each semantic object among multiple semantic objects, obtain the preset type weight boosting coefficient corresponding to the type identifier;

[0050] The text content of the semantic object is segmented into multiple terms;

[0051] An inverted index is constructed based on multiple terms, and the type weight boosting coefficient is associated with multiple terms in the inverted index.

[0052] It's important to note that during the construction of the inverted index, for each semantic object, the system first obtains a pre-defined type weight boosting coefficient corresponding to its type identifier (such as "title" or "abstract"). This coefficient represents the importance of that type in keyword retrieval (for example, words in the title may be more important than words in the body text). Next, the system performs word segmentation on the text content of the semantic object, breaking it down into multiple basic terms. When finally constructing the inverted index, the system not only records which semantic objects each term appears in (i.e., establishes a mapping from term to document), but also associates the previously obtained type weight boosting coefficient related to the semantic object type with all terms in that semantic object. This means that in the index structure, a term will carry different weight boosting information when it appears in different types of semantic objects.

[0053] Therefore, by associating and storing the type weighting coefficient with the term, a foundation is laid for subsequent keyword retrieval scoring. This allows terms appearing in important types (such as titles) to be given higher weight when calculating keyword relevance. Consequently, without changing the query, the ranking of documents containing important types of content is automatically improved, making the search results more biased towards key chapters and enhancing the search's guidance.

[0054] In one possible implementation, the vector index is an HNSW (Hierarchical Navigable SmallWorld) graph index, constructed based on multiple joint vectors of multiple semantic objects, including:

[0055] For each of the multiple joint vectors, determine the candidate nearest neighbor set of the joint vector, select the nearest neighbor vector from the candidate nearest neighbor set, establish the connection relationship between the joint vector and the nearest neighbor vector, and construct the HNSW graph index based on the connection relationship between each of the multiple joint vectors and its corresponding nearest neighbor vector.

[0056] Specifically, when selecting a nearest neighbor vector from the candidate nearest neighbor set, a preset composite distance calculation function is used to calculate the composite distance between the joint vector and each candidate nearest neighbor vector in the candidate nearest neighbor set. The nearest neighbor vector is selected from the candidate nearest neighbor set based on the composite distance. The composite distance calculation function is a weighted combination of the cosine distance between vectors and the type matching function. The type matching function takes a preset type guidance strength coefficient when the joint vector and the candidate nearest neighbor vector have the same type, and takes zero when the types are different.

[0057] It should be noted that this possible implementation specifically limits the data structure used for constructing the vector index to an HNSW graph index, and details its construction process, particularly the introduction of a type-guided mechanism. During construction, for each joint vector, the system finds a set of nearest neighbor vectors to establish connections in the graph. The key lies in the criteria for selecting nearest neighbors: it doesn't simply calculate the cosine distance between joint vectors (i.e., pure semantic similarity), but uses a pre-defined composite distance calculation function. This function is a weighted combination of the cosine distance between vectors and a type-matching function. The type-matching function returns a pre-defined positive value (type-guided strength coefficient) when the semantic objects from which two joint vectors originate have the same type identifier (e.g., both are "conclusion"); it returns zero when the types are different. Therefore, the composite distance considers both semantic similarity and type matching. Based on this composite distance, the system selects the final nearest neighbor vectors from the candidate nearest neighbor set for connection. By establishing such connections for all nodes (joint vectors) in the graph, the HNSW graph index is finally constructed.

[0058] Therefore, by constructing the HNSW graph index using a composite distance function, the type similarity information of semantic objects is pre-encoded into the index structure itself. This makes the system naturally inclined to navigate and find neighbors in regions of semantic objects of the same or similar types during subsequent graph search (retrieval) processes, thereby achieving a type clustering effect, improving retrieval efficiency, and potentially making the returned results more consistent with the inherent laws in terms of structural type.

[0059] In one possible implementation, the query intent includes intent category and confidence level, and the query entity set includes academic entities identified from the user's query command and the entity type of each academic entity; the query command is subjected to intent recognition to obtain the query intent and the query entity set, including:

[0060] The query command is preprocessed to obtain the preprocessed query text; wherein, the preprocessing includes at least one of the following: word segmentation, stop word filtering and spell correction;

[0061] Input the query text into a pre-trained academic query intent classifier to obtain the probability distribution output by the academic query intent classifier. The category with the highest probability in the probability distribution is taken as the intent category, and the probability value corresponding to the category with the highest probability is taken as the confidence level. The intent category includes at least one of the following: keyword retrieval, semantic retrieval, entity search, and review query.

[0062] Input the query text into the pre-trained academic named entity recognition model to obtain the academic entities in the query text and the entity type of each academic entity output by the academic named entity recognition model.

[0063] Construct a query entity set based on academic entities and entity types, where academic entities include at least one of the following: scholar's name, text title, and technical terminology.

[0064] It's important to note that the process begins with preprocessing the user's original query, including word segmentation, filtering out meaningless stop words, and correcting potential spelling errors, resulting in a clean and structured query text. Next, this query text is input into a pre-trained academic query intent classifier, which outputs a probability distribution for different intent categories. The system identifies the category with the highest probability (e.g., semantic retrieval or entity lookup) as the intent category for this query and uses this highest probability value as the confidence level for this identification. Simultaneously, the query text is input into another pre-trained academic named entity recognition model, which identifies and extracts academic entities (such as specific scholar names, paper titles, or technical terms) and the entity type of each entity from the query text. Finally, the system organizes these identified academic entities and their corresponding entity types to form a query entity set.

[0065] Therefore, by using a dedicated academic domain model (intent classifier and named entity recognition model) to perform deep analysis of user queries, not only are ambiguous query statements transformed into clear intent categories and confidence levels, but key academic entities and their types are also accurately extracted. This provides accurate and structured input information for subsequent retrieval filtering, strategy selection, and weight configuration.

[0066] In one possible implementation, the method further includes, before performing vector retrieval in the vector index based on the query intent and the set of query entities:

[0067] The vector index is filtered based on the query intent and the query entity set to obtain a candidate vector set;

[0068] Based on the query intent and query entity set, a vector retrieval is performed in the vector index to obtain a candidate semantic object set and the vector cosine similarity score of each candidate semantic object in the candidate semantic object set, including:

[0069] The query command is processed based on the query intent and the query entity set to obtain the query vector. Vector retrieval is then performed on the candidate vector set based on the query vector to obtain a preset number of joint vectors sorted from high to low similarity with the query vector.

[0070] Based on the semantic objects corresponding to the preset number of joint vectors, a set of candidate semantic objects is determined, and the vector cosine similarity score of each candidate semantic object in the set of candidate semantic objects is determined.

[0071] It's important to note that before performing vector retrieval based on the query intent and query entity set, the system first uses these two factors as conditions to filter the entire vector index, quickly eliminating a large number of obviously irrelevant vectors, thus obtaining a smaller set of candidate vectors. Subsequent vector retrieval operations are then performed on this reduced set of candidate vectors. Specifically, the system transforms the query instruction itself into a query vector based on the query intent and query entity set, and then uses this query vector to search the previously obtained set of candidate vectors (not the entire index) to find a predetermined number of joint vectors most similar to the query vector. The semantic objects corresponding to these joint vectors constitute the final set of candidate semantic objects, and their similarity value is the vector cosine similarity score.

[0072] Therefore, by employing a two-stage design of filtering followed by refined searching, retrieval efficiency is improved. Specifically, by utilizing high-level, structured information such as query intent and entity sets for rapid initial screening, the number of candidate objects requiring expensive vector similarity calculations is reduced. This allows the system to more quickly focus on the most likely range for accurate semantic matching within a large-scale index, balancing retrieval accuracy and speed.

[0073] In one possible implementation, the vector index is filtered based on the query intent and the query entity set to obtain a candidate vector set, including:

[0074] Combine the entity types of each academic entity into an entity type combination;

[0075] Using the combination of intent category and entity type as an index, the corresponding retrieval strategy descriptor is searched in the predefined retrieval strategy knowledge base. The retrieval strategy descriptor includes a list of semantic object types and metadata filter condition templates.

[0076] Based on the variable placeholders in the metadata filter condition template, find the corresponding entity values ​​from the query entity set, replace the variable placeholders with the entity values, and generate the metadata filter conditions.

[0077] The target type range is determined based on the list of semantic object types. The metadata filtering conditions and the target type range are applied to the vector index to obtain a joint vector that meets the metadata filtering conditions and the target type range, which is then used as a candidate vector set.

[0078] It's important to note that, firstly, the system combines the entity types of all academic entities in the query entity set into an entity type combination (e.g., "scholar's name + professional term"). Then, using the intent category and this entity type combination as the joint index key, it searches the retrieval strategy knowledge base for a matching retrieval strategy descriptor. This descriptor contains two key parts: a list of semantic object types, indicating which types of text should be prioritized in this query (e.g., focusing on citations in the "body text" when searching for scholars); and a metadata filtering condition template, which is a conditional expression with variable placeholders (e.g., {scholar's name}). Next, the system finds the actual values ​​(e.g., specific scholar names) from the query entity set, replaces the variable placeholders in the template, and generates an executable metadata filtering condition. Finally, the target type range determined by the semantic object type list, along with the generated metadata filtering condition, is applied to the metadata associated with the entire vector index, filtering out the joint vectors corresponding to those semantic objects that simultaneously meet the type requirements and specific metadata conditions (e.g., author is so-and-so). These filtered vectors constitute the candidate vector set.

[0079] Therefore, by using a strategy knowledge base, different combinations of query intents and entity types are mapped to specific retrieval strategies (including which chapters to focus on and which metadata to filter). This makes the filtering process no longer a simple keyword matching process, but a highly customized and precise screening based on domain knowledge. This can greatly improve the quality of the candidate set and ensure that subsequent vector semantic retrieval is carried out within a very relevant and pure range.

[0080] In one possible implementation, keyword retrieval is performed in the inverted index based on the query intent and the query entity set to obtain keyword relevance scores for each candidate semantic object in the candidate semantic object set, including:

[0081] A Boolean query tree is generated based on the query intent and the set of query entities. The Boolean query tree is then submitted to the inverted index, and keyword retrieval is performed to obtain a set of candidate semantic objects.

[0082] For each candidate semantic object in the candidate semantic object set, a keyword relevance score is calculated using a pre-defined BM25 (Best Match 25) variant algorithm; this includes:

[0083] For each candidate semantic object in the candidate semantic object set, obtain the type weight boosting coefficient corresponding to the candidate semantic object from the inverted index, and obtain the term frequency of each term in the candidate semantic object;

[0084] The term frequencies are weighted based on the type weighting factor to obtain the effective term frequencies, and the keyword relevance scores of candidate semantic objects are calculated based on the effective term frequencies.

[0085] It's important to note that, firstly, the system generates a structured Boolean query tree based on the query intent and the set of query entities (e.g., a certain term must appear, and a certain person's name cannot appear), and submits this query to the inverted index for execution, thus obtaining an initial set of candidate semantic objects. For each candidate semantic object in this set, the system uses a pre-defined variant of the BM25 algorithm to calculate its keyword relevance score. The key improvement of this variant algorithm is that it obtains the type weight boosting coefficient associated with the candidate semantic object from the inverted index. During the calculation, the algorithm obtains the term frequency of each matching term in the semantic object, and then uses this type weight boosting coefficient to weight these term frequencies to obtain an effective term frequency. Subsequent BM25 relevance score calculations will be based on this type-weighted effective term frequency, thus yielding the final keyword relevance score.

[0086] Therefore, by improving the classic BM25 algorithm, in calculating term importance, not only are factors such as term frequency and inverse document frequency considered, but a weight boosting based on semantic object type is also introduced. This ensures that matching terms appearing in important type parts (such as titles) contribute higher scores in keyword matching, making keyword retrieval results structurally sensitive and forming a more effective complement to purely semantic vector retrieval.

[0087] In one possible implementation, the query intent includes an intent category and a confidence level. Weight coefficients are determined from a pre-defined weight configuration table based on the query intent, including:

[0088] Based on the intent category, determine the high-confidence baseline weight vector and low-confidence baseline weight vector corresponding to the intent category from the weight configuration table;

[0089] If the confidence level is greater than or equal to the preset high confidence threshold, then the high confidence benchmark weight vector will be used as the weight coefficient.

[0090] If the confidence level is less than the preset low confidence threshold, the low confidence benchmark weight vector will be used as the weight coefficient.

[0091] If the confidence level is between the low confidence threshold and the high confidence threshold, then the weight coefficients are calculated using linear interpolation based on the high confidence benchmark weight vector, the low confidence benchmark weight vector, the confidence level, the high confidence threshold, and the low confidence threshold.

[0092] The weighting coefficients include keyword weight, semantic weight, and metadata weight, and the sum of keyword weight, semantic weight, and metadata weight is 1.

[0093] It should be noted that the system first reads two sets of baseline weight vectors from the weight configuration table based on the intent category (such as semantic retrieval): one set is a high-confidence baseline weight vector applicable to the intent, and the other set is a low-confidence baseline weight vector. Then, the system compares the actual identified confidence level with two preset thresholds (high-confidence threshold and low-confidence threshold): if the confidence level is very high (greater than or equal to the high threshold), the high-confidence weight vector is directly used; if the confidence level is very low (less than the low threshold), the low-confidence weight vector is directly used; if the confidence level is in the middle, a linear interpolation is performed between the high and low baseline weight vectors based on the confidence value to obtain a weight coefficient that is a smooth transition between the two.

[0094] Furthermore, this weight coefficient specifically includes three parts: keyword weight assigned to the keyword relevance score, semantic weight assigned to the vector cosine similarity score, and metadata weight assigned to the metadata matching, and the sum of the three is 1.

[0095] Therefore, when the system is very certain of the user's intent, it adopts a fixed weight optimized for that intent; when it is less certain, it adopts a more conservative or balanced weight; and when it is in an intermediate state, it smoothly transitions. This greatly improves the robustness of the system in handling fuzzy or ambiguous queries, making the final relevance score calculation more reasonable and flexible.

[0096] In one possible implementation, the cosine similarity score of the vectors is processed using weight coefficients to obtain the final relevance score of each candidate semantic object, including:

[0097] For each candidate semantic object in the candidate semantic object set, obtain the preset set of metadata attributes and the attribute weights corresponding to each metadata attribute;

[0098] For each metadata attribute in the metadata attribute set, obtain the attribute value corresponding to the metadata attribute in the metadata of the candidate semantic object, and obtain the entity value corresponding to the metadata attribute in the query entity set, and calculate the attribute matching value between the attribute value and the entity value;

[0099] The metadata matching score of the candidate semantic object is obtained by weighting and summing the matching values ​​of each attribute with their corresponding attribute weights.

[0100] Based on the keyword weight, semantic weight, and metadata weight in the weighting coefficients, the keyword relevance score, vector cosine similarity score, and metadata matching score of the candidate semantic object are weighted and summed to obtain the final relevance score of the candidate semantic object.

[0101] It should be noted that for each candidate semantic object, the system first defines a predefined set of metadata attributes (such as author, publication year, journal) and the attribute weight for each attribute. Then, for each metadata attribute in the set, the system performs two tasks: first, it retrieves the value of the attribute from the metadata of the candidate semantic object; second, it checks whether the user query mentions the entity value corresponding to that attribute (e.g., the query identifies the author entity). Next, it calculates the degree of matching between these two (attribute matching value), such as whether they are completely equal or partially matched. The metadata matching score of the candidate semantic object is obtained by multiplying the attribute matching values ​​of all attributes by their respective attribute weights and summing them. Finally, in the weighted fusion stage, the system uses the keyword weight, semantic weight, and metadata weight in the weight coefficients to perform weighted summation on the keyword relevance score, vector cosine similarity score, and the newly calculated metadata matching score of the candidate semantic object, thereby obtaining the final relevance score that integrates text content, semantics, type, and metadata information.

[0102] Therefore, structured metadata information (author, time, etc.) is explicitly incorporated into the comprehensive scoring system and assigned independent weights. This enables the retrieval system to not only match text content and semantics, but also accurately match metadata conditions that users may implicitly specify (such as finding articles published by a particular author in a specific journal in recent years). This achieves a deep integration of content retrieval and database query capabilities, greatly enhancing the ability to meet complex and precise retrieval needs.

[0103] In this embodiment, the text is first structured and parsed into semantic objects with type identifiers, thereby achieving fine-grained organization of the text content, facilitating retrieval for different chapters and improving processing efficiency. Furthermore, by fusing the text content vector and the type embedding vector into a joint vector and establishing an index, not only is semantic information preserved, but the structural features of the text are also incorporated, enhancing the expressive power of the index and thus improving the matching accuracy of the retrieval. During the retrieval stage, the retrieval results can be obtained by searching the previously constructed vector index according to the query instructions.

[0104] In summary, the optimization of the entire text retrieval process not only improves resource utilization but also significantly enhances retrieval efficiency and accuracy, thereby improving the user experience.

[0105] This paper provides an overview of the text retrieval method illustrated in the embodiments of this application, focusing on improving resource utilization and supporting multiple retrieval scenarios. A detailed explanation follows:

[0106] 1. Overall Overview of the Plan

[0107] The technical solution illustrated in this application aims to meet the needs of large-scale academic paper data management and querying by optimizing the implementation of academic paper text data retrieval methods. This solution is mainly described in detail from two aspects: optimization of the data preprocessing workflow and optimization of the hybrid retrieval algorithm.

[0108] 1.1 Text Data Preprocessing Optimization

[0109] To fully leverage the clear structural characteristics of academic papers and reduce the storage space required for indexing, the solution first optimizes text data preprocessing techniques. By segmenting the paper text into semantic paragraphs such as title, abstract, introduction, main body, and conclusion, instead of the traditional method of directly segmenting text paragraphs, the storage requirements for inverted indexes and vector indexes are significantly reduced.

[0110] 1.2 Hybrid Search Algorithm Optimization

[0111] To expand the retrieval capabilities supported by existing text retrieval algorithms, this method designs retrieval algorithms that support multiple retrieval scenarios, thereby supporting diverse paper data retrieval needs.

[0112] Furthermore, the retrieval algorithm in this method optimizes the subsequent processing of keyword retrieval results and vector retrieval results. By dynamically weighting and fusing keyword relevance scores and vector cosine similarity scores, the algorithm achieves an understanding of query intent and adjusts the weight ratio between keyword matching and semantic matching, thereby improving retrieval accuracy and efficiency.

[0113] 2. Technical Solution Flow

[0114] The overall solution overview proposes improving resource utilization and supporting diverse retrieval needs by optimizing academic paper text data retrieval methods. The following sections will elaborate on the specific implementation process based on two aspects: text data preprocessing optimization and hybrid retrieval algorithm optimization. The core of the solution includes two main modules: data processing and indexing, and hybrid querying. These two modules handle data indexing and querying respectively to meet the management and querying needs of large-scale paper data. The following will combine... Figure 2A and Figure 2B The process details the operation mechanism of these two modules.

[0115] 2.1 Text Data Processing and Indexing

[0116] The purpose of this module is to transform unstructured raw academic paper text into structured index data for efficient retrieval, thereby providing a high-quality, low-redundancy data foundation for subsequent hybrid retrieval. Specific steps include:

[0117] Data cleaning: For user-uploaded paper texts, the system first performs a data cleaning step. This step not only removes HTML / XML tags, garbled characters, and special characters, but also, based on the characteristics of academic texts, further filters out meaningless headers, footers, page numbers, journal copyright information, and other noisy data to ensure the purity of the text content for subsequent processing.

[0118] Text normalization: The cleaned text data undergoes comprehensive normalization processing, including converting English characters to lowercase, removing Chinese and English punctuation marks, and converting full-width characters to half-width characters, thereby achieving the standardization of text units.

[0119] Rich Text Parsing and Multi-Granularity Semantic Object Generation: Addressing the characteristics of academic papers' standardized formatting and clear logical structure, the method described in this application employs a combination of rule-based parsing and pre-trained models for deep content extraction. Specifically, the system comprehensively utilizes layout analysis, regular expressions, and natural language processing tools, such as NLTK (Natural Language Toolkit) and Spacy (a high-performance natural language processing Python library focused on industrial applications), to automatically identify and extract metadata (title, author, year, journal, etc.) and core content paragraphs (abstract, introduction, body, conclusion) from academic papers. Each extracted unit is constructed as a semantic object, and each object not only contains text content but also its type (text content). ), (type represents the type identifier of the semantic object, which includes title, abstract, introduction, body text and conclusion), and the paper ID to which it belongs ( This includes metadata and other information. After this step, each academic paper text will be modeled and represented by the system using a structured object set rich in textual data and additional information.

[0120] Inverted Index Construction: After generating multi-granularity semantic objects, the system enters the inverted index construction stage. This application's embodiments design and implement an enhanced inverted index construction method based on semantic object type weights. This method quantifies the type attribute of semantic objects into specific weight boosting factors and directly applies them to the index construction process, thereby solidifying prior knowledge in the academic field within the index structure. In specific implementation, the system predefines a static type weight boosting coefficient for each semantic object type. This coefficient is a set of numerical parameters pre-determined through offline analysis and experiments, based on the relative importance of different types of semantic objects in academic retrieval. In the specific implementation, the system assigns a weight boosting coefficient of 2.0 to the "Title" type semantic object, 1.5 to the "Abstract" and "Conclusion" types, 1.2 to the "Introduction" type, and a baseline coefficient of 1.0 to the "Main Text" type. When building an inverted index using an index library (such as Lucene), the system does not process all text content equally. For each semantic object to be indexed, the system first obtains its corresponding type weight boosting coefficient based on its type identifier. Subsequently, when adding the text content of this semantic object to the inverted index, the system uses this coefficient as the base weight multiplier for all terms in that object. From a technical implementation perspective, this is equivalent to explicitly setting a document-level weight boosting coefficient, which is equal to the type weight boosting coefficient, for the indexed document corresponding to this semantic object when calling the file addition interface of the index library.

[0121] This operation ensures that terms originating from high-importance semantic objects (such as titles and abstracts) have their base weight in the inverted index pre-enhanced during index building. The technical effect of this design is that it enables subsequent keyword retrieval operations to calculate keyword similarity scores (…). When calculating the keyword relevance score, the underlying calculation model already incorporates a quantitative assessment of academic importance. This means that when a term appears in both the title and the body text, the term in the title receives a higher base weight during the indexing process, resulting in a higher score in the keyword relevance score. The computation inherently possesses higher scoring potential, which is reflected in the "keyword retrieval and vector retrieval" steps described later. This is mainly achieved by regulating the effective term frequency t_effective, and by differentiating at the vector representation level (type-aware vector augmentation): through a supervised training process, a unique "type signature" (i.e., type embedding vector) is learned for each type and fused with the text content vector; and by differentiating at the index topology level (type similarity-guided nearest neighbor selection): when constructing the HNSW graph, the nearest neighbor selection function is modulated to give higher connection probabilities between vectors of the same type.

[0122] Vector Index Construction: This step primarily involves vectorizing the text content of the aforementioned semantic objects and constructing an efficient HNSW to support vector similarity retrieval. When constructing the vector index, the system designs and implements a joint vector indexing method based on type embedding fusion. This method injects category semantic information into the vector representation through a deterministic and reproducible mathematical process, thereby achieving accurate retrieval and routing within the index structure. Type-aware vector augmentation is the cornerstone of this method, and its implementation includes an offline type embedding table training phase and an online vector fusion phase.

[0123] During the training phase, the system first initializes a trainable type embedding query table, which is a table with one dimension. The matrix, where This represents the total number of semantic object types supported by the system. The system represents the preset dimension of the type embedding vectors. Subsequently, the system jointly fine-tunes the text embedding model and the type embedding table on a next-sentence prediction or semantic matching task constructed from academic paper data. During this process, each training sample consists of text content and its corresponding semantic object type label. The model's learning objective is that when the text embedding vectors and the correct type embedding vectors are fused according to predetermined rules, the resulting joint vector exhibits optimal performance on the target task used for model training / fine-tuning. Through this process, the type embedding table is trained to convergence, where each row of vectors encodes the deep semantic role of the corresponding semantic object type in the academic retrieval task.

[0124] During the online index building phase, for each semantic object to be indexed, the system performs the following fusion operation to generate its joint vector representation: First, the system uses a fine-tuned text embedding model to convert the text content of the semantic object into a high-dimensional text content vector. Secondly, based on the type identifier (type) of the semantic object, the system retrieves the corresponding type embedding vector from the pre-trained type embedding lookup table. Finally, the system uses a concatenation function to... and The fusion, e.g., `Concat( [1, 0, 1], [1, 1, 1]) = [1, 0, 1, 1, 1, 1]`, generates the final joint vector corresponding to the text content of the semantic object. .

[0125] After generating all joint vectors, the system will construct a unified HNSW (Hierarchical Navigable Small World) graph index based on this set of joint vectors. During the index construction process, the system further implements a type-guided nearest neighbor selection strategy. The core of this strategy lies in modifying the nearest neighbor selection function of the HNSW algorithm; that is, in the HNSW algorithm, for each newly inserted vector... From candidate nearest neighbor set When selecting the final connection, the system redefines the distance calculation function as follows: , in The cosine similarity between vectors The preset type guidance strength coefficient (typical value is 0.1). For type matching functions (such as when) The value is 1 when the nearest neighbor is selected and 0 otherwise. This distance function ensures that vectors of the same type will be selected during the nearest neighbor selection process. Distance reduction increases the probability of being selected as a nearest neighbor under the same vector cosine similarity, thereby ensuring that in the unified HNSW index, semantic objects of the same type naturally form densely connected sub-communities, while objects of different types only maintain necessary semantic connectivity connections.

[0126] Index storage: After the inverted index and vector index are serialized, they are persistently stored in the file system or database. Once stored, subsequent retrieval processes can directly load these index files for queries.

[0127] 2.2 Dynamically Weighted Hybrid Retrieval

[0128] The purpose of this module is to process user queries and return the most relevant academic content. Its technical significance lies in understanding the user's query intent, intelligently integrating the results of keyword matching and semantic matching, and ultimately providing accurate search results. Specific steps include:

[0129] Query preprocessing: Automated preprocessing of the user-inputted raw query statement, including word segmentation, stop word filtering, spell correction, and other operations, to prepare for subsequent in-depth analysis.

[0130] Academic Query Parsing and Intent Recognition: The system inputs the preprocessed query into a pre-trained academic query intent classifier. This classifier employs a lightweight BERT (Bidirectional Encoder Representations from Transformers) model architecture based on an attention mechanism and is trained on a manually labeled academic query dataset. It can identify whether the user's query is biased towards "keyword retrieval," "semantic retrieval," "entity search," or "review query" (I, I∈{I...}). keyword I semantic I entity I survey (where I represents the intent category, which can belong to any of the following intents: keyword search intent, semantic search intent, entity search intent, summary search intent, etc.) and output the classification results and confidence scores. While recognizing the user's query intent, the system concurrently invokes a Named Academic Recognition (NER) model to accurately identify academic entities such as scholar names, paper titles, and technical terms in the query to construct a query entity set. .

[0131] Query strategy execution and retrieval strategy generation: The system is based on the intent category I and confidence level output during the query parsing phase. and entity set The system dynamically generates a structured retrieval strategy for the user's current query input through a defined algorithm. This process does not rely on fuzzy heuristics but is based on a complete, predefined retrieval strategy knowledge base. Specifically, this knowledge base is essentially a two-dimensional decision table, with row indexes representing all intent types supported by the system and column indexes representing all possible entity type combinations E. comb Each cell (I, Ecomb) stores a corresponding retrieval strategy descriptor. The descriptor is a structure containing three fields: A well-defined list of semantic object types defines the target scope of the language objects being retrieved. A template string for metadata filtering conditions, containing placeholders for variables to be populated (e.g., ...). ); : Retrieval pattern identifier, such as (Exact match) or (Semantic focus).

[0132] During algorithm execution, the system will perform the following algorithm steps to generate a retrieval strategy:

[0133] ① Strategy search and matching: using As input, where This is the set of entities identified during the query phase. The system first starts from... Extract entity type combinations Used as a lookup key and located in the strategy knowledge base. To ensure completeness, the knowledge base is designed according to the MECE principle (mutually exclusive, collectively exhaustive), providing completeness for all... All combinations have predefined strategies. In the rare case of an undefined combination, the system defaults to a conservative strategy. For all types, Empty;

[0134] ②Conditional instantiation: System traversal Each variable placeholder in the entity set, and in the entity set The corresponding entity value is searched and replaced to generate a specific, executable metadata filtering condition. (For example, templates) In entity set There exists When, it will be instantiated as );

[0135] ③ Confidence level adaptation: The system adapts the confidence level... For the generated filter conditions Softening is performed using a predefined softening function; that is, for string matching conditions, when... At that time, the system will use the exact equality operator. Replace with similarity matching function For numerical matching conditions, exact matches are replaced with range matches (e.g., ...). After executing the algorithm, the system will generate a retrieval strategy. The code is compiled into two execution plans through syntax transformation: for inverted indexes, the plan is compiled into a Boolean query tree, the structure of which is... ,in, From targeting The composition of the query terms for the fields. Depend on and Jointly build (such as) .

[0136] Furthermore, ① in type IN(A), A comes from... Take it out, as mentioned earlier. ① A clearly defined list of semantic object types, defining the target scope of the language objects for this retrieval; ② The B in AND B is actually... As mentioned earlier How is it generated based on template string instantiation? For vector indexing, the plan is compiled into a filtering expression, the core of which is to perform a filtering process based on the template string before vector retrieval. and Perform a pre-filtering operation to narrow down the candidate vector set; the expression is: The direct logical representation is used to pre-filter candidate vectors before vector retrieval operations, while... It is converted into a vector ID candidate range to ensure that vector retrieval is performed only in the vector set of the target type.

[0137] Keyword retrieval and vector retrieval: After obtaining the compiled execution plan, the system executes keyword retrieval and vector retrieval operations in parallel. In keyword retrieval, the system submits the compiled Boolean query tree to the inverted index engine (such as Lucene) for execution and integrates the document-level type weight factors assigned during the inverted index construction phase into the keyword relevance score by modifying the internal implementation of the BM25 algorithm. In the calculation process, specifically, the system employs a type-weighted variant of the BM25 algorithm. This algorithm redefines the term frequency (tf) component in the standard BM25 formula, and in calculating a term... In the document When considering the weighted contribution in the algorithm, the document is introduced. Corresponding type weight boosting factor (Determined during the inverted index construction phase, as mentioned above with the title 2.0, etc., type weight) This directly amplifies the effectiveness of term frequencies in highly important semantic objects (such as titles and abstracts). This means that when the same term appears in the title, its contribution to the final BM25 score S_bm25 is significantly higher due to the amplification effect of W_type=2.0 than its contribution when it appears in the main text (W_type=1.0). This is achieved by using [a modulating factor] to adjust the effective term frequencies. .

[0138] The calculation formula is:

[0139] ;

[0140] in, Original word frequency, and These are the free parameters for BM25. For document length, This represents the average document length. Meanwhile, in the Boolean query tree... Part of the process is executed in the initial stage of the keyword retrieval process to quickly exclude documents that do not meet the type and metadata filtering conditions. The specific execution process in the vector retrieval step is a sophisticated two-stage pipeline. In the first stage, the system compiles the filtering expressions... and type range It operates on the entire HNSW vector index, quickly retrieving a set of semantic object IDs that meet the criteria through efficient metadata indexing of the vector database. In the second phase, the system uses this ID set. The system performs HNSW approximate nearest neighbor search. To ensure a balance between search quality and efficiency, the system... The size of the collection is dynamically set by HNSW. Parameters are used to ensure a high vector recall rate across different retrieval scales, i.e., when hour, ;when hour, ;when hour, =400. The final vector retrieval operation returns a list of semantically most similar objects in the pre-filtered set, along with their vector cosine similarity scores. .

[0141] Multi-factor dynamic weighted fusion: For the candidate results returned in parallel, this application embodiment uses a multi-factor dynamic weighted model for calculation, and the final relevance score is calculated as follows:

[0142]

[0143] in, For keyword similarity score, The vector cosine similarity score is used. This is the matching score between the metadata of the candidate semantic object and the academic entity in the query. In the formula above, the weights... It is not a fixed value, but depends on the constraints. Under these conditions, the specific values ​​of the three weights are dynamically adjusted by the output of the query intent classifier, enabling the system to adaptively respond to diverse user query intents. For example, if the classifier determines that the intent is "keyword retrieval," the weights are automatically increased. If a scholar entity with high confidence is identified, then increase the confidence level. .

[0144] Furthermore, in this embodiment of the application, the metadata matching score It is a normalization to The weighted sum of matching scores for multiple metadata attributes within a range is used to measure the relationship between the metadata of the candidate semantic object and the set of academic entities identified in the query. The degree of matching is calculated using the following formula:

[0145]

[0146] in, It is a collection of metadata attributes used for matching, including but not limited to information such as author, year, and title. It is an attribute The weights, and satisfying The constraints. It is an attribute The matching function returns a similarity value of 0 (no match), 1 (complete match), or a value in between. For example, for the author attribute, if querying the entity set... The included scholar names and candidates If the fields are completely identical If the similarity calculated using the normalized Levenshtein distance exceeds the threshold (0.8), then... Otherwise, it is 0 (this rule also applies to the calculation of attributes such as title and journal name); for the year attribute, if the query contains the year entity... And if the year of the candidate object is equal to the year of the query, then If in Within the range, then Otherwise, it is 0 (this rule also applies to the calculation of other numeric metadata attributes).

[0147] Results sorting and return: The system sorts all candidate semantic objects according to their final scores from highest to lowest, and returns results after identifying duplicates. The most relevant results are output to the user as the final search results.

[0148] 3. Comprehensive Implementation of the Technical Solution

[0149] The technical solution of this application optimizes the academic paper text data retrieval method, improving resource utilization and retrieval accuracy and efficiency to meet the management and query needs of large-scale academic paper text data. The comprehensive implementation process of this solution will be described in detail below, in conjunction with the foregoing sections, and further technical details will be added.

[0150] 3.1 System Overall Architecture

[0151] The technical solution of this application embodiment constitutes a complete intelligent system for academic paper retrieval through the synergy of the aforementioned data processing, index construction, and hybrid retrieval modules. This system is the physical implementation of the above-mentioned methods in this application embodiment. Its core is a software system that integrates business logic, data processing, and retrieval capabilities, rather than a single vector database. It covers the complete process from data preprocessing and query understanding to result fusion.

[0152] This system utilizes a vector database as one of its underlying text data indexing and storage components for efficient storage and retrieval of semantic vectors. It also constructs the inverted index of the method described in this application's embodiments by additionally integrating the Apache Lucene text processing toolkit (Apache Lucene, a mature, high-performance full-text search engine library, is integrated into the vector database engine layer) into the vector database engine layer, thereby enabling support for keyword retrieval. For example... Figure 3 and Figure 4 The system's layered architecture design and internal component design were identified.

[0153] The method described in this application divides the system into a service layer and a data layer. The service layer is further divided into an external access layer and a business logic layer. The external access layer is responsible for receiving user operation requests and forwarding them to the corresponding module in the business logic layer for processing based on the type of operation. The business logic layer consists of a query processing module and a data update module, responsible for querying relevant text data and updating index data files and metadata files, respectively. The data storage layer is responsible for indexing and persistently storing the text data.

[0154] 3.2 Semantic Object Generation for Academic Papers

[0155] Building upon the process described in section 2.1, the implementation of the method shown in this application further optimizes the semantic object generation strategy. The system leverages the inherent structural features of academic papers, achieving high-precision content deconstruction by fusing rule templates and pre-trained models. For example, for PDF format papers, the system first uses PDF parsing tools (such as Apache PDFBox) to extract text flow and layout information. Then, it uses a Transformer-based sequence labeling model (such as LayoutLM) to identify the logical structure of the text flow (title, author, abstract, introduction, body text, conclusion, etc.). Finally, it divides the content into corresponding semantic objects according to predefined rules. Each semantic object is assigned a type identifier and its associated metadata information and document unique identifier.

[0156] 3.3 Fusion Mechanism Based on Multi-Factor Dynamic Weighting

[0157] A key aspect of the method described in this application is the need to implement an adaptive weighted fusion system based on query intent parsing for academic text retrieval scenarios. This mechanism transforms the abstract intent of a user's query into specific retrieval parameters through a coherent computation process, thereby improving the accuracy of result ranking.

[0158] The mechanism begins with academic queries and intent recognition. User queries are first processed by an academic query intent classifier based on a Transformer architecture, which uses a BERT-based model fine-tuned on a large-scale query dataset (such as manually labeled training data). The output of the model's last layer is then processed... After the function is transformed into a probability distribution, the system determines the category with the highest probability as the predicted intent of this query. (like (etc.) and take its probability value as the confidence level. Meanwhile, the user's query text is input in parallel into an academic named entity recognition (NER) model based on a BiLSTM-CRF architecture to accurately identify the set of academic entities contained in the query. And its type (including scholar, institution, paper title, conference journal, year, etc.).

[0159] After obtaining the intent classification results and confidence levels, the system enters the dynamic weight decision-making stage. Weight coefficients. The value is determined by the intent category. and confidence level Decisions made jointly and subject to constraints. The system's built-in weight configuration strategy defines two sets of baseline weight vectors for each intent. , respectively corresponding to high confidence levels State and low confidence state And the confidence level in the middle range is achieved through linear interpolation. The system employs a smooth transition during the decision-making process to form a complete dynamic decision-making model. Specifically, the baseline weight vectors for different intent types within this system are shown in Table 1 below.

[0160] Table 1

[0161]

[0162] When confidence level At that time, this system calculates the final weights using linear interpolation. The interpolation formula is:

[0163]

[0164] Finally, the system synthesizes and ranks the relevance scores. Each candidate semantic object receives a keyword relevance score. Vector cosine similarity score and metadata matching score After calculation, the weight system formula for dynamic decision-making is as follows:

[0165]

[0166] A linearly weighted sum is performed to generate the final relevance score for the candidate object. All candidate objects are then sorted in descending order based on their final scores, generating an ordered list of search results which is returned to the user.

[0167] In summary, the method described in this application provides an innovative solution to the key shortcomings of existing technologies in the storage and retrieval of academic paper text data, significantly improving system performance and user experience. Compared to existing technologies, it has the following advantages:

[0168] 1. Higher resource utilization rate:

[0169] Advantages of semantic paragraph segmentation: This segmentation method significantly reduces the number of text paragraphs and the required storage space. By utilizing the structural characteristics of academic texts to divide the text into five semantic paragraphs, the excessive storage space required by inverted indexes and vector indexes can be avoided, significantly improving storage efficiency while meeting the retrieval needs of academic papers.

[0170] 2. More comprehensive and precise search support:

[0171] Advantages of multi-query scenario support technology: By leveraging an academic query intent classifier, the traditional text hybrid retrieval method is optimized, extending the query scenarios to multiple query scenarios such as keyword retrieval, semantic retrieval, entity query, and review query, meeting the diverse retrieval needs of users when searching for academic papers; Advantages of dynamic weighted hybrid query technology: The optimized dynamic weighted hybrid query algorithm automatically identifies the bias type of the query through the query classifier and then adaptively adjusts the value of the weight factor, making full use of the query results of keyword retrieval and vector similarity retrieval, balancing the query needs under different query inputs, and improving the accuracy and efficiency of hybrid retrieval.

[0172] Figure 5 An embodiment of the present application is shown, such as a text retrieval device. Figure 5 As shown, the device 50 includes:

[0173] The acquisition module 501 is used to acquire the original text, parse the text, and generate multiple semantic objects carrying type identifiers, wherein the type identifiers include at least one of the following: title, abstract, introduction, body text, and conclusion;

[0174] The execution module 502 is used to input the text content of the semantic object into the pre-trained text embedding model for each semantic object, obtain the text content vector output by the text embedding model, look up the corresponding type embedding vector from the pre-trained type embedding lookup table according to the type identifier of the semantic object, concatenate the text content vector and the type embedding vector to obtain a joint vector, and construct a vector index based on multiple joint vectors of multiple semantic objects.

[0175] The execution module 502 is also used to receive the user's query instruction, perform a search in the vector index according to the query instruction, and obtain the search results.

[0176] In this embodiment, the text is first structured and parsed into semantic objects with type identifiers, thereby achieving fine-grained organization of the text content, facilitating retrieval for different chapters and improving processing efficiency. Furthermore, by fusing the text content vector and the type embedding vector into a joint vector and establishing an index, not only is semantic information preserved, but the structural features of the text are also incorporated, enhancing the expressive power of the index and thus improving the matching accuracy of the retrieval. During the retrieval stage, the retrieval results can be obtained by searching the previously constructed vector index according to the query instructions.

[0177] In summary, the optimization of the entire text retrieval process not only improves resource utilization but also significantly enhances retrieval efficiency and accuracy, thereby improving the user experience.

[0178] This application provides a network device 60, such as... Figure 6 As shown, the network device 60 includes a processor 601, a memory 602, and a program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, it implements the steps of a text retrieval method as shown in the above embodiment.

[0179] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of a text retrieval method as shown in the above embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0180] This application also provides a computer program product, including computer instructions. When executed by a processor, the computer instructions implement the steps of the text retrieval method shown in the above embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0181] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0183] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A text retrieval method, characterized in that, The method includes: Obtain the original text, parse the text, and generate multiple semantic objects carrying type identifiers, wherein the type identifiers include at least one of the following: title, abstract, introduction, body text, and conclusion; For each semantic object, the text content of the semantic object is input into a pre-trained text embedding model to obtain the text content vector output by the text embedding model. The corresponding type embedding vector is searched from the pre-trained type embedding lookup table according to the type identifier of the semantic object. The text content vector and the type embedding vector are concatenated to obtain a joint vector. A vector index is constructed based on multiple joint vectors of multiple semantic objects. Receive the user's query instruction, perform a search in the vector index according to the query instruction, and obtain the search results; The vector index is a hierarchical navigable small-world (HNSW) graph index, constructed based on multiple joint vectors of multiple semantic objects, including: For each of the plurality of joint vectors, a candidate nearest neighbor set is determined, a nearest neighbor vector is selected from the candidate nearest neighbor set, a connection relationship is established between the joint vector and the nearest neighbor vector, and the HNSW graph index is constructed based on the connection relationship between each of the plurality of joint vectors and its corresponding nearest neighbor vector. Specifically, when selecting the nearest neighbor vector from the candidate nearest neighbor set, a preset composite distance calculation function is used to calculate the composite distance between the joint vector and each candidate nearest neighbor vector in the candidate nearest neighbor set. The nearest neighbor vector is selected from the candidate nearest neighbor set based on the composite distance. The composite distance calculation function is a weighted combination of the cosine distance between vectors and a type matching function. The type matching function takes a preset type guidance strength coefficient when the joint vector and the candidate nearest neighbor vector have the same type, and takes zero when the types are different.

2. The method according to claim 1, characterized in that, The search is performed in the vector index according to the query instruction, and the search results include: The query command is subjected to intent recognition to obtain the query intent and query entity set; Based on the query intent and the query entity set, a vector retrieval is performed in the vector index to obtain a candidate semantic object set and the vector cosine similarity score of each candidate semantic object in the candidate semantic object set. Based on the query intent, weight coefficients are determined from a preset weight configuration table, and the weight coefficients are used to process the vector cosine similarity score to obtain the final relevance score of each candidate semantic object. The candidate semantic objects are sorted according to the final relevance score to obtain a sorting result, and the search result is returned based on the sorting result.

3. The method according to claim 2, characterized in that, In the case of constructing a vector index based on multiple joint vectors of multiple semantic objects, the method further includes: constructing an inverted index based on multiple semantic objects; In the case that vector retrieval is performed in the vector index according to the query intent and the query entity set to obtain a candidate semantic object set and a vector cosine similarity score of each candidate semantic object in the candidate semantic object set, the method further includes: performing keyword retrieval in the inverted index according to the query intent and the query entity set to obtain a keyword relevance score of each candidate semantic object in the candidate semantic object set; The final relevance score of each candidate semantic object is obtained by weighting the vector cosine similarity score using the weight coefficients. This includes weighting and fusing the keyword relevance score and the vector cosine similarity score of each candidate semantic object using the weight coefficients.

4. The method according to claim 3, characterized in that, An inverted index is constructed based on multiple semantic objects, including: For each semantic object among the multiple semantic objects, obtain a preset type weight boosting coefficient corresponding to the type identifier; The text content of the semantic object is segmented to obtain multiple terms; The inverted index is constructed based on the multiple terms, and the type weight boosting coefficient is associated with and stored in the inverted index with the multiple terms.

5. The method according to claim 2 or 3, characterized in that, The query intent includes intent category and confidence level, and the query entity set includes academic entities identified from the user query instruction and the entity type of each academic entity; The query command is subjected to intent recognition to obtain the query intent and query entity set, including: The query command is preprocessed to obtain preprocessed query text; wherein the preprocessing includes at least one of the following: word segmentation, stop word filtering, and spell correction; The query text is input into a pre-trained academic query intent classifier to obtain the probability distribution output by the academic query intent classifier. The category with the highest probability in the probability distribution is taken as the intent category, and the probability value corresponding to the category with the highest probability is taken as the confidence level. The intent category includes at least one of the following: keyword retrieval, semantic retrieval, entity search, and review query. The query text is input into a pre-trained academic named entity recognition model to obtain the academic entities in the query text and the entity type of each academic entity output by the academic named entity recognition model. The query entity set is constructed based on the academic entities and the entity types, wherein the academic entities include at least one of the following: scholar's name, text title, and professional terminology.

6. The method according to claim 5, characterized in that, Before performing a vector retrieval in the vector index based on the query intent and the query entity set, the method further includes: The vector index is filtered based on the query intent and the query entity set to obtain a candidate vector set; Based on the query intent and the query entity set, a vector retrieval is performed in the vector index to obtain a candidate semantic object set and the vector cosine similarity score of each candidate semantic object in the candidate semantic object set, including: The query instruction is processed based on the query intent and the query entity set to obtain a query vector. Vector retrieval is then performed on the candidate vector set based on the query vector to obtain a preset number of joint vectors sorted from high to low similarity to the query vector. Based on the semantic objects corresponding to the preset number of joint vectors, the candidate semantic object set is determined, and the vector cosine similarity score of each candidate semantic object in the candidate semantic object set is determined.

7. The method according to claim 6, characterized in that, Based on the query intent and the query entity set, the vector index is filtered to obtain a candidate vector set, including: The entity types of each of the academic entities are combined into an entity type combination; Using the combination of intent category and entity type as an index, the corresponding retrieval strategy descriptor is searched in a predefined retrieval strategy knowledge base, wherein the retrieval strategy descriptor includes a list of semantic object types and a metadata filtering condition template; Based on the variable placeholders in the metadata filtering condition template, the corresponding entity values ​​are found from the query entity set, and the variable placeholders are replaced with the entity values ​​to generate metadata filtering conditions. The target type range is determined based on the list of semantic object types. The metadata filtering conditions and the target type range are applied to the vector index to obtain a joint vector that meets the metadata filtering conditions and the target type range, which is used as the candidate vector set.

8. The method according to claim 4, characterized in that, Based on the query intent and the query entity set, keyword retrieval is performed in the inverted index to obtain the keyword relevance score of each candidate semantic object in the candidate semantic object set, including: A Boolean query tree is generated based on the query intent and the query entity set. The Boolean query tree is then submitted to the inverted index, and keyword retrieval is performed to obtain the candidate semantic object set. For each candidate semantic object in the candidate semantic object set, a preset BM25 variant algorithm is used to calculate the keyword relevance score; including: For each candidate semantic object in the candidate semantic object set, the type weight boosting coefficient corresponding to the candidate semantic object is obtained from the inverted index, and the term frequency of each term in the candidate semantic object is obtained; The term frequencies are weighted based on the type weighting factor to obtain effective term frequencies, and the keyword relevance score of the candidate semantic object is calculated based on the effective term frequencies.

9. The method according to claim 3, characterized in that, The query intent includes intent category and confidence level. Weight coefficients are determined from a preset weight configuration table based on the query intent, including: Based on the intent category, determine the high-confidence baseline weight vector and low-confidence baseline weight vector corresponding to the intent category from the weight configuration table; If the confidence level is greater than or equal to the preset high confidence threshold, then the high confidence benchmark weight vector is used as the weight coefficient. If the confidence level is less than a preset low confidence threshold, then the low confidence benchmark weight vector is used as the weight coefficient. If the confidence level is between the low confidence threshold and the high confidence threshold, then the weight coefficient is calculated by linear interpolation based on the high confidence benchmark weight vector, the low confidence benchmark weight vector, the confidence level, the high confidence threshold, and the low confidence threshold. The weighting coefficients include keyword weight, semantic weight, and metadata weight, and the sum of the keyword weight, the semantic weight, and the metadata weight is 1.

10. The method according to claim 9, characterized in that, The cosine similarity score of the vectors is processed using the weight coefficients to obtain the final relevance score of each candidate semantic object, including: For each candidate semantic object in the candidate semantic object set, a preset set of metadata attributes and the attribute weights corresponding to each metadata attribute are obtained; For each metadata attribute in the metadata attribute set, obtain the attribute value corresponding to the metadata attribute in the metadata of the candidate semantic object, and obtain the entity value corresponding to the metadata attribute in the query entity set, and calculate the attribute matching value between the attribute value and the entity value; The metadata matching score of the candidate semantic object is obtained by weighting and summing the matching values ​​of each attribute with the corresponding attribute weights. Based on the keyword weight, semantic weight, and metadata weight in the weighting coefficients, the keyword relevance score, vector cosine similarity score, and metadata matching score of the candidate semantic object are weighted and summed to obtain the final relevance score of the candidate semantic object.

11. A text retrieval device, characterized in that, The device includes: The acquisition module is used to acquire the original text, parse the text, and generate multiple semantic objects carrying type identifiers, wherein the type identifiers include at least one of the following: title, abstract, introduction, body, and conclusion; The execution module is used to input the text content of the semantic object into a pre-trained text embedding model for each semantic object, obtain the text content vector output by the text embedding model, look up the corresponding type embedding vector from the pre-trained type embedding lookup table according to the type identifier of the semantic object, concatenate the text content vector and the type embedding vector to obtain a joint vector, and construct a vector index based on multiple joint vectors of multiple semantic objects. The execution module is also used to receive the user's query instruction, perform a search in the vector index according to the query instruction, and obtain the search results; The vector index is a hierarchical navigable small-world (HNSW) graph index, constructed based on multiple joint vectors of multiple semantic objects, including: For each of the plurality of joint vectors, a candidate nearest neighbor set is determined, a nearest neighbor vector is selected from the candidate nearest neighbor set, a connection relationship is established between the joint vector and the nearest neighbor vector, and the HNSW graph index is constructed based on the connection relationship between each of the plurality of joint vectors and its corresponding nearest neighbor vector. Specifically, when selecting the nearest neighbor vector from the candidate nearest neighbor set, a preset composite distance calculation function is used to calculate the composite distance between the joint vector and each candidate nearest neighbor vector in the candidate nearest neighbor set. The nearest neighbor vector is selected from the candidate nearest neighbor set based on the composite distance. The composite distance calculation function is a weighted combination of the cosine distance between vectors and a type matching function. The type matching function takes a preset type guidance strength coefficient when the joint vector and the candidate nearest neighbor vector have the same type, and takes zero when the types are different.

12. A network device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of a text retrieval method as claimed in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a text retrieval method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of a text retrieval method as described in any one of claims 1 to 10.

Images