Semi-structured text multi-level information retrieval method

Abstract

The invention discloses a semi-structured text multilevel information retrieval method, and belongs to the field of information retrieval and data mining. According to the method, firstly, semi-structured text data is analyzed, a filtering index is constructed for structured metadata, and a semantic vector index and a keyword index are constructed for unstructured text content in parallel; during retrieval, firstly, rapid coarse screening is carried out by utilizing structured conditions so as to reduce the range; then semantic retrieval and keyword retrieval are executed in parallel in the candidate set, and a semantic correlation score and a keyword matching score are obtained respectively; and then carrying out weighted fusion on the two types of scores through a dynamic fusion layer, wherein weight parameters can be dynamically called according to a user identifier or a query scene. According to the method, the balance of retrieval efficiency and precision is realized, the problem of static stiffness of the traditional mixed retrieval weight is solved, the retrieval strategy can be adaptively adjusted according to implicit feedback of the user, and the personalized intelligent level of a retrieval system is improved.

Description

Technical Field

[0001] This invention relates to the field of information retrieval and data mining, and in particular to a method for multi-level information retrieval of semi-structured text, which can realize intelligent and adaptive multi-level information retrieval of semi-structured text data. Background Technology

[0002] In today's big data environment, massive amounts of information exist in semi-structured form and continue to grow. This type of data contains both structured attributes and unstructured text content. For example, in e-commerce, product information includes structured fields such as price, brand, and category, as well as unstructured text such as product titles, descriptions, and user reviews; in enterprise knowledge bases, documents include attributes such as author, department, and creation date, as well as core text such as report text and technical solutions; in academic databases, papers include attributes such as publication year, journal, and author, as well as abstracts and full text. How to efficiently and accurately retrieve the information needed by users from this type of semi-structured text data is a core challenge faced by many application systems.

[0003] Existing search technologies can be mainly divided into the following categories, but each has its own limitations:

[0004] The first category is retrieval based on structured attributes. These methods utilize database query languages ​​(such as SQL) or filters to locate data through exact matching or range filtering. Their advantages are speed and predictable results, but their disadvantages are extremely obvious: they are completely unable to understand and process the rich semantic information contained in unstructured text. When a user's query intent needs to be expressed through text content, such as with a mobile phone with a good camera, simple structured retrieval is powerless.

[0005] The second category is keyword-based full-text search. These technologies (such as Lucene and Elasticsearch based on inverted indexes) segment and index the text content, measuring the relevance between documents and queries by calculating statistical measures such as term frequency and inverse document frequency (e.g., TF-IDF, BM25 models). Their advantages include mature technology, high efficiency, and good matching of precise terms and phrases. However, their essence is "lexical matching," which suffers from a "semantic gap": they cannot identify synonyms (e.g., "laptop" and "handheld computer"), semantically related words (e.g., "Apple" and "iPhone"), and have a weak ability to understand the intent of natural language queries (e.g., keyword search for "sci-fi movies suitable for children" struggles to effectively link "children," "teenagers," and "sci-fi movies").

[0006] The third category is semantic-based vector retrieval. With the development of deep learning, semantic retrieval techniques based on pre-trained language models (such as BERT and Sentence-BERT) have been applied. This type of method converts text into high-dimensional vectors (Embeddings) and measures semantic relevance by calculating the cosine similarity between vectors. Its core advantage lies in its ability to deeply understand semantics, overcome the "lexical semantic gap," and return semantically related results with different word choices. However, it also has shortcomings: firstly, for "hard" information that requires precise matching, such as model numbers, codes, and proper nouns, its accuracy may not be as good as keyword retrieval; secondly, its retrieval performance heavily depends on the domain relevance of the training corpus.

[0007] To overcome the limitations of single technologies, hybrid retrieval strategies have emerged, attempting to combine the advantages of multiple retrieval technologies. However, existing hybrid retrieval schemes still suffer from the following prominent problems:

[0008] 1. Rigid Integration Methods: Most solutions employ simple linear weighted summation or fixed priority concatenation (e.g., keywords first, then semantics, or vice versa). Their weighting parameters (such as the weight ratio of semantic score to keyword score) are typically static, globally preset values ​​obtained through offline experiments, failing to adapt to the dynamic needs of different query scenarios. For example, for the query "iPhone13 Pro Max 256GB Blue," accurate product model matching (keyword advantage) should dominate; while for "an elegant gift for a lady," semantic understanding (the association between gift, elegance, and lady) is more crucial. Static weighting cannot achieve this kind of scenario-adaptive adjustment.

[0009] 2. Lack of Adaptability and Personalization: Once deployed, existing systems retain a fixed retrieval model (including mixed weights). They cannot self-optimize based on actual user interactions. Different user groups, and even the same user, may have different retrieval preferences in different scenarios, but the system cannot perceive and adapt to these differences, resulting in a poor long-term personalized experience in search results.

[0010] 3. Insufficient utilization of feedback: Although some systems have introduced feedback mechanisms, they mostly rely on user-displayed ratings or annotations. This type of data is costly to acquire, sparse, and may contain subjective biases. Furthermore, there is a lack of effective mechanisms to utilize implicit feedback data generated by natural user interactions (such as clicks and dwell time) for real-time, automated optimization of the retrieval model's mixed weights.

[0011] Therefore, there is an urgent need for a semi-structured text retrieval method that can organically integrate structured filtering, semantic matching, and keyword matching, and can utilize implicit feedback to optimize its internal fusion strategy in real time and adaptively, thereby achieving breakthroughs in retrieval efficiency, accuracy, and personalized experience. Summary of the Invention

[0012] To address key issues in existing hybrid retrieval schemes, such as rigid fusion methods, lack of adaptability and personalization, and insufficient feedback utilization, this invention provides a multi-level information retrieval method for semi-structured text. This method achieves an adaptive retrieval process integrating efficient initial screening, precise fine screening, dynamic fusion, and closed-loop optimization. It can solve the intelligent retrieval challenges of semi-structured text (such as military intelligence reports, policy documents, and scientific and technological literature, including structured metadata such as author, date, and security classification, and unstructured text such as title and body text) in military, government, and academic fields.

[0013] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0014] A semi-structured text multi-level information retrieval method includes the following steps:

[0015] Step 1: Parse the semi-structured text data, separate the structured metadata from the unstructured text content, build an aggregated filtering index for the structured metadata, and build a semantic vector index and a keyword index for the unstructured text content;

[0016] Step 2: Receive user queries. First, perform a coarse screening based on the structured metadata index to obtain an initial candidate set. Then, for the documents in the initial candidate set, execute the semantic retrieval path and the keyword retrieval path in parallel to obtain the semantic relevance score and keyword matching score for each document. Finally, through a dynamic fusion layer, use adaptively allocated weight parameters to perform weighted fusion of the semantic relevance score and keyword matching score of the same document to obtain the final ranking score. Return the search results based on the final ranking score.

[0017] Step 3: Collect implicit interaction data of users with search results; construct positive and negative sample pairs based on implicit interaction data, and calculate the ranking loss under the current weight parameters; based on the ranking loss, perform real-time incremental updates on the weight parameters of the dynamic fusion layer, associate the updated weight parameters with the corresponding user's identifier and store them for use in the user's subsequent search process.

[0018] Furthermore, in step 1, the semantic vector index is constructed as follows:

[0019] The pre-trained language model is fine-tuned using domain-related corpora to obtain a domain-enhanced semantic encoding model;

[0020] Unstructured text content is encoded into semantic vectors using a domain-enhanced semantic coding model;

[0021] The semantic vectors are stored in a vector database that supports approximate nearest neighbor search.

[0022] Furthermore, in step 1, the keyword index is constructed as follows:

[0023] Based on a domain dictionary and a stop word list, unstructured text content is segmented.

[0024] An inverted index is constructed based on the word segmentation results, and statistical information of the terms is calculated to support the BM25 relevance scoring model.

[0025] Furthermore, in step 2, a coarse screening is performed based on the index of structured metadata, specifically as follows:

[0026] Parse user queries and identify the structured constraints within them;

[0027] The structured constraints are combined into a Boolean query, and documents are filtered in the aggregated filtering index to form an initial candidate set.

[0028] Further, in step 2, the semantic retrieval pathway specifically involves: using a domain-enhanced semantic coding model to map the user query and candidate document content into semantic vectors respectively; calculating the cosine similarity between the query vector and the document vector, and normalizing it to obtain a semantic relevance score.

[0029] Furthermore, in step 2, the keyword retrieval pathway specifically involves: segmenting the user query and candidate document content into words; using the BM25 statistical relevance model to calculate the matching degree between the query and the document, and then normalizing it to obtain the keyword matching score.

[0030] Furthermore, in step 2, the specific method of weighted fusion is as follows:

[0031] Final_Score = α Score_semantic + β Score_keyword

[0032] Where Final_Score is the final ranking score, Score_semantic is the semantic relevance score, Score_keyword is the keyword matching score, α and β are weight parameters, and α+β=1;

[0033] The weight parameters are dynamically obtained from a preset weight configuration library based on the user identifier or query scenario type.

[0034] Furthermore, in step 3, positive and negative sample pairs are constructed based on implicit interaction behavior data, specifically as follows:

[0035] Documents where users click and stay for more than the first threshold, or where a conversion occurs, are marked as positive samples;

[0036] In the returned search results list, documents that rank higher than or close to positive samples but have not been clicked, or whose dwell time after being clicked is less than the second threshold, are marked as negative samples.

[0037] Furthermore, in step 3, the ranking loss is calculated using a hinge loss function based on pairwise learning. For a pair of positive samples P and negative samples N, the loss is... The calculation method is as follows:

[0038]

[0039] in, For preset boundary values, and These are the final ranking scores for positive sample P and negative sample N, respectively.

[0040] Furthermore, in step 3, the weight parameters of the dynamic fusion layer are updated incrementally in real time using the online gradient descent method, specifically as follows:

[0041] Calculate the ranking loss with respect to the weight parameters gradient ;

[0042] Weight parameters From the original value Updated to :

[0043]

[0044] in, The learning rate;

[0045] according to , weight parameters The value is updated to :

[0046] .

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0048] 1. Balancing search efficiency and accuracy: The "structured coarse screening" filters out irrelevant data first, significantly reducing the overhead of subsequent complex semantic and keyword calculations. The "dual-path fine screening" ensures high accuracy in core document matching, balancing semantic breadth and keyword precision.

[0049] 2. Achieved true adaptive fusion: Through the linkage of the "dynamic fusion layer" and the "feedback optimization layer", the drawbacks of static and rigid mixed retrieval weights are broken, so that the contribution ratio of semantics and keywords can be dynamically adjusted according to the real feedback of users, adapting to the personalized needs of different queries and different users.

[0050] 3. A closed-loop system for sustainable evolution has been formed: by making full use of implicit feedback as a low-cost data source, the core parameters are continuously optimized through an online learning mechanism, enabling the system to have the ability to iterate itself and become more accurate with use, which significantly improves the long-term user experience and the level of system intelligence. Attached Figure Description

[0051] Figure 1 This is a flowchart of a semi-structured text multi-level information retrieval method provided in an embodiment of the present invention.

[0052] Figure 2 This is a schematic diagram illustrating the implementation principle of a semi-structured text multi-level information retrieval method provided in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and do not constitute a limitation thereof.

[0054] A multi-level information retrieval method for semi-structured text is proposed, the core of which lies in constructing a cascaded processing framework comprising a structured coarse screening layer, a dual-path fine screening layer, a dynamic fusion layer, and a feedback optimization layer. The method first parses the semi-structured text data, constructing a filtering index for structured metadata and a semantic vector index and keyword index in parallel for unstructured text content. During retrieval, a rapid coarse screening is first performed using structured conditions to narrow down the scope; then, semantic retrieval and keyword retrieval are performed in parallel within the candidate set, obtaining semantic relevance scores and keyword matching scores respectively; subsequently, the two scores are weighted and fused through a dynamic fusion layer, where the weight parameters can be dynamically retrieved based on user identification or query scenario. Furthermore, by collecting implicit feedback behaviors such as user clicks and dwell time, sample pairs are constructed and ranking loss is calculated. Personalized fusion weights for users are incrementally updated in real time using an online learning approach, forming a closed loop of "retrieval-feedback-optimization".

[0055] like Figure 1 As shown, the method specifically includes the following steps:

[0056] S1, Data Preprocessing and Index Building: Parsing semi-structured text data to separate its structured metadata from unstructured text content. Building an Elasticsearch-based aggregated filtering index for structured metadata (such as author, editor, creation date, publishing organization, document type, etc.); and for unstructured text content, building a domain-enhanced semantic vector index and a TF-IDF-based bag-of-words model in parallel.

[0057] S2, Dual-path fine screening and dynamic fusion: Within the document set obtained from the coarse screening, two retrieval paths are launched in parallel to target the core textual intent of the query:

[0058] S2.1, Semantic Retrieval Path: A BERT series model fine-tuned on domain-related corpora is used to map the query text and candidate document content to the same high-dimensional semantic space, generating deep semantic vectors. A semantic relevance score is obtained by calculating the cosine similarity between the query vector and the document vector.

[0059] S2.2 Keyword Retrieval Path: A domain-based bag-of-words model is constructed using the TF-IDF algorithm combined with the domain bag-of-words approach to calculate keyword matching degree between the query and candidate documents. The BM25 statistical relevance model is used to calculate the keyword matching score.

[0060] S2.3, Dynamic Fusion Layer: This layer incorporates a weighted adaptive allocation module that weights and fuses the semantic and keyword scores of the same document to generate a final ranking score. The key is that the weights are not fixed values. The retrieval system has a weight configuration library to store and call dynamic weight parameters for different user identifiers or query scenario types. In the absence of user-personalized weights, an initial weight is assigned based on a lightweight query scenario classifier to reflect the completeness of the retrieval system.

[0061] S3, Closed-loop optimization based on implicit feedback: To achieve adaptive evolution of the retrieval model, this method introduces a feedback optimization layer to form a closed loop of "retrieval-feedback-optimization".

[0062] S3.1 Implicit Feedback Signal Acquisition: During the interaction between the user and the search results, the retrieval system seamlessly collects various implicit behavioral data, including but not limited to: click sequences on search results, dwell time on the result details page, skipping behavior on results, and subsequent conversion behaviors such as downloading and saving.

[0063] S3.2, Feedback Signal Analysis and Loss Calculation: Documents where users engage in positive interactions (e.g., clicking and staying for a relatively long time) are considered positive samples, while documents ranked high but ignored or quickly skipped are considered negative samples. Based on these sample pairs (positive and negative samples), the ranking loss under the current dynamic fusion layer weights is calculated. Through a ranking-based loss function, the final score of positive samples is encouraged to be higher than that of negative samples.

[0064] S3.3, Real-time Incremental Parameter Update: Using incremental gradient descent, the weight parameters of the dynamic fusion layer are updated based on the loss calculated in S3.2. Fine-tuning was performed, with updates aimed at minimizing losses, namely improving ranking performance to align with users' actual preferences.

[0065] S3.4, Personalized weights stored in the application: The updated weight parameters ( , The system associates the corresponding user identifier with the weight configuration library. When this user initiates a search, the search system will prioritize calling their personalized weight parameters for score fusion, thereby achieving personalized adaptation of the search strategy.

[0066] This method achieves a balance between retrieval efficiency and accuracy, overcomes the problem of static and rigid weights in traditional hybrid retrieval, and can adaptively adjust the retrieval strategy based on implicit user feedback, thereby improving the personalized and intelligent level of the retrieval system.

[0067] Here is a more specific example:

[0068] A semi-structured text multi-level information retrieval method, such as Figure 1 and Figure 2 As shown, it includes the following steps:

[0069] S1, Data preprocessing and index building.

[0070] This step is the preparation phase, aimed at establishing an efficient, multi-granular indexing system for semi-structured text data. Taking a document database as an example, each document may contain the following fields:

[0071] Structured metadata includes document number, security classification, publishing organization, document type, creation / receipt date, associated geographic coordinates, etc.

[0072] Unstructured text content: document title, body text, appendix, abstract, etc.

[0073] S1.1, Data parsing and field separation.

[0074] Use the appropriate parser (such as a parser for JSON or XML formats, or one that extracts data from a relational database) to read the data. Explicitly identify fields such as security classification, issuing unit, document type, and date as structured fields; identify fields such as title and body text as unstructured text fields.

[0075] S1.2, Construct a structured metadata index.

[0076] Leveraging Elasticsearch's aggregation and filtering capabilities, index all structured fields. For example, create a keyword index for confidential fields to support precise filtering; create a date index for creation dates to support range queries; and create multi-field indexes (text and keyword) for publishing organizations to support aggregate analysis. The core purpose of this indexing is to achieve millisecond-level multi-dimensional combined filtering.

[0077] S1.3, Construct a domain-enhanced semantic vector index.

[0078] 1. Domain model fine-tuning: Select a pre-trained language model (such as bert-base-chinese), and use a domain-specific corpus for supervised or unsupervised continued pre-training or downstream task fine-tuning to obtain a domain-enhanced semantic encoding model (hereinafter referred to as Domain-BERT). This step enables the model to have a deeper understanding of domain terms.

[0079] 2. Generate document vectors: Use the Domain-BERT model to encode the core text content of each document (usually the concatenation of the title and the first N valid characters of the body) into a vector of a fixed dimension (e.g., 768 dimensions). To improve efficiency, the hidden state of the [CLS] token or the average of the segmented text encodings can be used.

[0080] 3. Build a vector database: Store the generated document vectors and their corresponding document IDs in an Elasticsearch plugin that supports vector retrieval, and build an efficient approximate nearest neighbor (ANN) index for fast similarity search of high-dimensional vectors.

[0081] S1.4, Build a domain-optimized keyword index.

[0082] 1. Domain dictionary integration: Import the domain-specific dictionary to ensure that terms such as "drone" and "UAV" can be correctly segmented and recognized. At the same time, a stop word list can be constructed to filter out function words such as "de" and "le" that have no actual retrieval meaning, as well as overly general and non-discriminatory words in the domain.

[0083] 2. Build an inverted index: Use Elasticsearch to build a traditional keyword inverted index for the same text content (title + body). When building the index, use the domain dictionary for word segmentation and calculate statistical information such as TF-IDF. This index is used to support keyword relevance scoring based on the BM25 and its variant algorithms.

[0084] 至此,为同一份数据建立了三种视图的索引:用于快速过滤的结构化索引、用于深度语义匹配的向量索引、用于精确词汇匹配的关键词索引。

[0085] S2, Multi-level hybrid retrieval and dynamic fusion.

[0086] When the user sets the data range and enters a query on the front-end interface, the retrieval system executes the following process online:

[0087] S2.1, Structured rough screening.

[0088] 1. Query Parsing: First, parse the query statement to identify the structured constraints. Using rule templates or a lightweight NER model, identify:

[0089] Time range: "Since 2023" -> Structured conditions: Creation date >= "2023-01-01 ~ present".

[0090] Publisher: "Published by xxx unit" -> Structured condition: Publisher = "xxx unit" (or match the list of subordinate units).

[0091] Document type / content hint: "Data Analysis" -> Structured conditions: Document type = "Data Briefing" (associated through a preset mapping table).

[0092] “xxx training” is identified as text intent and left for further processing by the fine screening layer.

[0093] 2. Efficient Filtering: The extracted structured conditions are combined into an Elasticsearch Boolean query to retrieve data from the structured metadata index built in step S1. This step does not perform relevance scoring; it only performs efficient document ID filtering. Assuming the database has 10 million documents, this step can quickly narrow the scope to tens of thousands or even thousands of documents that only meet the above conditions, forming an initial candidate set. This greatly reduces the burden on subsequent computationally intensive retrieval steps.

[0094] S2.2, dual-path parallel fine screening layer.

[0095] The initial candidate set is obtained through structured coarse screening. Then, targeting the core textual intent of the query. Two retrieval pathways are launched in parallel to screen and score candidate documents from two dimensions: semantic understanding and word matching.

[0096] 1. Semantic Retrieval Pathway: This path aims to capture the deep semantic relevance between queries and documents, overcoming the lexical gap problem. Its core is to utilize domain-enhanced pre-trained language models to map text into a high-dimensional semantic space and calculate it through vector similarity.

[0097] Semantic vector generation: for user queries and candidate documents The core document content uses a semantic encoding model finely tuned on a domain corpus. (e.g., Domain-BERT) generates its corresponding semantic vector representation.

[0098]

[0099]

[0100] in, and yes 3D real vectors (e.g.) ).Model The parameters are obtained through domain adaptation training in the offline phase, enabling it to have a stronger understanding of specific domain terms and contexts.

[0101] Semantic relevance score: for each candidate document Calculate its document vector With query vector The cosine similarity between them serves as the original semantic relevance measure. :

[0102]

[0103] in," " represents the vector dot product, This represents the L2 norm of a vector.

[0104] Score normalization: In order to map semantic similarity scores to a unified standard To integrate with keyword scores, a minimum-maximum normalization method is used, scaling the range based on the score range within the current candidate set.

[0105]

[0106] in, and They represent exist The minimum and maximum values ​​within the range.

[0107] The final result That is, a document With query The degree of semantic relevance. The higher the score, the more semantically relevant the document and the query are, even if they use different surface terms.

[0108] 2. Keyword Retrieval Pathway: This path aims to accurately match keywords, terms, and entities in queries and documents, ensuring the accuracy of retrieval of hard information (such as model numbers, codes, and proper nouns). Its core is based on calculating term-level matching degrees using statistical language models.

[0109] Query and document representation: First, the query is segmented using an integrated domain dictionary. and documents The text is segmented into words, resulting in separate sets of terms. After stop word filtering, a list of terms is obtained. .

[0110] BM25 Relevance Calculation: For each candidate document The keyword matching score is calculated using the BM25 formula. The BM25 algorithm comprehensively considers term frequency (TF), inverse document frequency (IDF), and document length normalization. Its formula is as follows:

[0111]

[0112] in:

[0113] : query One of the terms.

[0114] It is a term In the document Frequency of occurrence (word frequency) in the text.

[0115] :document The length (in terms of the number of terms).

[0116] : The average length of all documents in the dataset.

[0117] and b: Adjustable parameters (usually) ), b controls the word frequency saturation rate, and b controls the strength of document length normalization.

[0118] : term The inverse document frequency is calculated using the following formula: Where N is the total number of documents in the dataset, It contains terms The number of documents. This gives rare terms higher weight.

[0119] Fraction normalization:

[0120] Similarly, normalize the original BM25 scores to Intervals, to ensure consistency with the semantic score scale:

[0121]

[0122] After normalization, The range of values ​​is The higher the score, the better the keyword match between the document and the query.

[0123] Through the two pathways described above, for each document in the initial candidate set Two complementary fractions are calculated in parallel:

[0124] and These two scores will be used as inputs and fed into a subsequent dynamic fusion layer for adaptive weighting to generate the final ranking criteria.

[0125] S2.3, Dynamic Fusion Layer.

[0126] 1. Weight Acquisition: For the current user (assuming user ID is...) The query retrieves the weight configuration library. On the first retrieval, the library may not contain the configuration. The records are then assigned a set of default global weights based on the query scenario classifier (a lightweight text classification model that determines whether the query tends towards "precise targeting" or "fuzzy semantics"). For example: , If it exists in the library Historical weights are used first, for example: , .

[0127] 2. Score Fusion: For each document in the initial candidate set... Calculate its final score:

[0128]

[0129] in .use The personalized weighting means that the retrieval system is more inclined to trust the results of semantic retrieval (weight 0.7), which may be because the user's historical behavior indicates that he pays more attention to conceptual associations than literal matching.

[0130] 3. Result sorting and return: Based on All candidate documents are sorted in descending order, and the top K (e.g., K=20) documents, along with their metadata, summaries, and other information, are returned to the user's front-end interface for display.

[0131] S3, closed-loop optimization based on implicit feedback.

[0132] After users receive the results list, their interaction behavior is collected in real time to optimize the retrieval system. Strategies such as "micro-batch processing" or "updating after a certain amount of data accumulation" can be employed to ensure the stability of the retrieval system and reduce computational overhead.

[0133] S3.1, Implicit feedback signal acquisition.

[0134] Suppose that among the 20 returned results, the user performed the following operations in sequence:

[0135] Clicked on the document ranked 3rd. It paused for 120 seconds and was saved (strong positive signal).

[0136] Quickly skipped (staying for <2 seconds) the 1st and 2nd ranked documents , (Potentially difficult negative samples).

[0137] Clicked on the document ranked 5th. However, it only stayed for 5 seconds before returning to the list (weak positive or neutral signal).

[0138] The retrieval system records these behavioral sequences and their timestamps without the system noticing.

[0139] S3.2, Feedback signal analysis and loss calculation.

[0140] After collecting the implicit user behavior sequences, the retrieval system needs to transform these raw signals into supervisory signals that can be used for model optimization. The core of this step is to construct high-quality sample pairs and calculate the ranking loss.

[0141] 1. Rules for constructing sample pairs:

[0142] Selection of positive samples (P): Documents in which the user has engaged in explicit positive interaction are selected. The rules are defined as follows: If a user clicks on a document and meets any of the following conditions, the document is marked as a positive sample:

[0143] a) The duration of stay exceeds the preset strong positive threshold. (For example, Second).

[0144] b) After clicking, further conversion behaviors such as saving, downloading, or forwarding are triggered.

[0145] According to this rule, in the example of S3.1, the document (Click, stay for 120 seconds, favorite) was identified as a strong positive sample.

[0146] Selection of Difficult Negative Samples (N): To effectively optimize the ranking effect, especially to improve the accuracy of top results, this method focuses on selecting difficult negative samples. The definition rule is as follows: In the Top-K results list returned by the current retrieval, documents that rank higher than or close to a certain positive sample but have been explicitly ignored or quickly skipped by the user. Specifically:

[0147] a) Documents that did not receive a click and whose ranking in the list is higher than that of a positive sample.

[0148] b) A click occurs, but the dwell time is extremely short (e.g., less than T_negative=2 seconds), and it is considered a "misclick" or an unsatisfactory document.

[0149] According to this rule, in the example, the documents ranked 1st and 2nd and quickly skipped... and Selected for positive samples Difficult negative samples. Document (Click but stay for 5 seconds) Due to its blurry signal, it will not be included in this update to avoid introducing noise.

[0150] 2. Design and calculation of the loss function:

[0151] The goal is to maximize the final ranking score of the positive samples. Significantly higher than negative samples This method adopts the pairwise learning to rank paradigm, and preferentially uses hinge loss as the loss function for online optimization because it is more sensitive to sample pairs near the boundary and is suitable for incremental updates.

[0152] For a sample pair consisting of positive sample P and negative sample N its losses The calculation is as follows:

[0153]

[0154] in:

[0155] It is a preset boundary value greater than 0 (for example, =0.1). This parameter defines the expected minimum difference between positive and negative sample scores. (Introduction) It can enhance the robustness of the model and prevent scores from being too close.

[0156]

[0157]

[0158] In a single user session, multiple valid [databases] may be generated. Yes. Assume this session is based on positive samples. Two negative sample pairs were generated: and The total loss for this retrieval task is then... The sum of losses for all sample pairs:

[0159]

[0160] The total loss quantifies the current dynamic fusion weights. The loss value indicates the degree of inconsistency between the ranking results and the user's implicit preferences. The larger the loss value, the more serious the deviation between the current weight-based ranking (negative samples ranked ahead of or too close to positive samples) and the user's true preferences.

[0161] S3.3, real-time incremental parameter updates.

[0162] The goal of this step is to calculate the loss based on S3.2. Fine-tuning the weight parameters of the dynamic fusion layer in a lightweight and efficient manner and This minimizes the loss, thus making the sorting more in line with user preferences.

[0163] 1. Gradient calculation:

[0164] An online gradient descent method is used. First, the total loss is calculated. Regarding weight parameters The gradient (partial derivative).

[0165] because (satisfy ), just for Optimize.

[0166] For a single sample pair loss Its effect on The gradient derivation is as follows:

[0167]

[0168] Derivation Explanation:

[0169] when When the loss is 0, the gradient is also 0, and no update is needed.

[0170] When the above conditions are not met .

[0171] Will Substituting the values ​​and taking the derivative, we obtain the above equation. It is the difference between the semantic score and the keyword score of the positive sample. These are the corresponding differences between negative samples. The gradient direction indicates the direction to reduce loss. The direction that should be adjusted.

[0172] Total loss right gradient It is the sum of the gradients of each sample pair:

[0173]

[0174] 2. Weight parameter update:

[0175] After obtaining the gradient, use the following formula to adjust the weights. Perform incremental updates:

[0176]

[0177] in:

[0178] The learning rate is a small positive number (e.g., This is used to control the step size of each update, ensuring the smoothness of the update.

[0179] The old weights used in this search (e.g.) (0.7).

[0180] This is the updated weight.

[0181] Due to constraints ,renew for:

[0182]

[0183] The direction of the update is to reduce losses. That is, let the positive samples The final score relative to the negative samples , Higher. Assuming this update makes... It increased slightly from 0.70 to 0.72. The value decreased from 0.30 to 0.28. This means that the retrieval system learned from this interaction that, for the user... For this type of query, semantic scores should be relied upon more heavily. Although the changes are small, the retrieval system can robustly learn the optimal personalized fusion strategy through the continuous accumulation of massive user sessions.

[0184] S3.4, Personalized Weight Storage and Application.

[0185] Updated weights With user identifier This is associated with the contextual features of the current query (e.g., categorized as "fuzzy semantic query") and persistently stored in the weight configuration library. When a user... The next time a search is performed, the retrieval system will directly load and integrate this updated set of personalized weights, thereby achieving real-time adaptive and personalized search strategies. With continuous accumulation of interactions, the retrieval system "sculpts" the most suitable mixed weights for different users and even different query types, achieving a closed-loop evolution that becomes "smarter with use." In practical applications, the updated weights can be further refined... Apply simple boundary constraints (e.g., limit to) Within the range), to ensure the robustness of the retrieval system and avoid weight loss due to a few abnormal feedbacks.

[0186] As can be seen from the above specific embodiments, the present invention effectively solves the problems pointed out in the background art through the coordinated work of the structured coarse screening layer, the dual-path fine screening layer, the dynamic fusion layer, and the feedback optimization layer:

[0187] 1. Balance between efficiency and accuracy: The pre-screening of structured data significantly improves overall efficiency; the parallel processing of semantic and keyword data ensures high accuracy and robustness in core text matching.

[0188] 2. Dynamic Adaptive Fusion: The fusion weights are no longer statically configured, but dynamic parameters that can learn and adjust online based on real-time user feedback, thus achieving flexibility and personalization of the fusion strategy.

[0189] 3. Closed-loop continuous optimization: By leveraging implicit feedback, a low-cost and high-coverage data source, to drive the evolution of the core parameters of the retrieval system, a self-improving intelligent closed loop is formed, which significantly improves the long-term performance and user experience of the retrieval system.

[0190] In summary, this invention efficiently and flexibly utilizes structured metadata for initial data screening to quickly narrow down the search scope and improve the overall efficiency of the retrieval system. During the fine screening stage, to overcome the limitations of a single retrieval model, a hybrid retrieval mechanism is implemented that simultaneously considers semantic understanding depth and keyword matching accuracy. This invention breaks through the limitations of traditional static models, utilizing low-cost, easily accessible implicit user feedback data to achieve real-time, personalized adaptive optimization of the core parameters of the retrieval model (especially the hybrid weights), enabling the retrieval system to become increasingly intelligent with use.

[0191] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for retrieving multi-level information from semi-structured text, characterized in that, Includes the following steps: Step 1: Parse the semi-structured text data, separate the structured metadata from the unstructured text content, build an aggregated filtering index for the structured metadata, and build a semantic vector index and a keyword index for the unstructured text content; Step 2: Receive user queries. First, perform a coarse screening based on the structured metadata index to obtain an initial candidate set. Then, for the documents in the initial candidate set, execute the semantic retrieval path and the keyword retrieval path in parallel to obtain the semantic relevance score and keyword matching score for each document. Finally, through a dynamic fusion layer, use adaptively allocated weight parameters to perform weighted fusion of the semantic relevance score and keyword matching score of the same document to obtain the final ranking score. Return the search results based on the final ranking score. Step 3: Collect implicit interaction data of users with search results; construct positive and negative sample pairs based on implicit interaction data, and calculate the ranking loss under the current weight parameters; based on the ranking loss, perform real-time incremental updates on the weight parameters of the dynamic fusion layer, associate the updated weight parameters with the corresponding user's identifier and store them for use in the user's subsequent search process.

2. The method for semi-structured text multi-level information retrieval according to claim 1, characterized in that, In step 1, the semantic vector index is constructed as follows: The pre-trained language model is fine-tuned using domain-related corpus to obtain a domain-enhanced semantic encoding model; Unstructured text content is encoded into semantic vectors using a domain-enhanced semantic coding model; The semantic vectors are stored in a vector database that supports approximate nearest neighbor search.

3. The method for semi-structured text multi-level information retrieval according to claim 1, characterized in that, In step 1, the keyword index is constructed as follows: Based on a domain dictionary and a stop word list, unstructured text content is segmented. An inverted index is constructed based on the word segmentation results, and statistical information of the terms is calculated to support the BM25 relevance scoring model.

4. The semi-structured text multi-level information retrieval method according to claim 1, characterized in that, In step 2, a coarse screening is performed based on the index of structured metadata. The specific method is as follows: Parse user queries and identify the structured constraints within them; The structured constraints are combined into a Boolean query, and documents are filtered in the aggregated filtering index to form an initial candidate set.

5. The method for semi-structured text multi-level information retrieval according to claim 1, characterized in that, In step 2, the semantic retrieval pathway specifically involves: using a domain-enhanced semantic coding model to map user queries and candidate document content into semantic vectors; calculating the cosine similarity between the query vector and the document vector, and normalizing it to obtain a semantic relevance score.

6. The method for semi-structured text multi-level information retrieval according to claim 1, characterized in that, In step 2, the keyword retrieval pathway specifically involves: segmenting the user query and candidate document content into words; using the BM25 statistical relevance model to calculate the matching degree between the query and the document, and then normalizing it to obtain the keyword matching score.

7. The method for semi-structured text multi-level information retrieval according to claim 1, characterized in that, In step 2, the specific method of weighted fusion is as follows: Final_Score = α Score_semantic + β Score_keyword Where Final_Score is the final ranking score, Score_semantic is the semantic relevance score, Score_keyword is the keyword matching score, α and β are weight parameters, and α+β=1; The weight parameters are dynamically obtained from a preset weight configuration library based on the user identifier or query scenario type.

8. The method for semi-structured text multi-level information retrieval according to claim 1, characterized in that, In step 3, positive and negative sample pairs are constructed based on implicit interaction behavior data, specifically as follows: Documents where users click and stay for more than the first threshold, or where a conversion occurs, are marked as positive samples; In the returned search results list, documents that rank higher than or close to positive samples but have not been clicked, or whose dwell time after being clicked is less than the second threshold, are marked as negative samples.

9. A semi-structured text multi-level information retrieval method according to claim 1 or 8, characterized in that, In step 3, the ranking loss is calculated using a hinge loss function based on pairwise learning. For a pair of positive samples P and negative samples N, the loss is... The calculation method is as follows: ； in, For preset boundary values, and These are the final ranking scores for positive sample P and negative sample N, respectively.

10. A semi-structured text multi-level information retrieval method according to claim 1, characterized in that, In step 3, the weight parameters of the dynamic fusion layer are updated incrementally in real time using the online gradient descent method, specifically as follows: Calculate the ranking loss with respect to the weight parameters gradient ; Weight parameters From the original value Updated to : ； in, The learning rate; according to , weight parameters The value is updated to : 。

Images