An archive level weight-based semantic retrieval sorting method and related device

Abstract

The application provides a semantic retrieval sorting method based on archive level weight, and related equipment. The method comprises: obtaining a natural language query sentence input by a user, and obtaining a plurality of to-be-retrieved archives, each to-be-retrieved archive comprising a plurality of level texts, the plurality of level texts comprising an archivist level text, a catalog level text, a volume level text, a file level text, and a page level text; determining a level semantic similarity between the natural language query sentence and the archivist level text, the catalog level text, the volume level text, the file level text, and the page level text of each to-be-retrieved archive; performing weighted summation processing on the level semantic similarity of each to-be-retrieved archive according to a plurality of level weights corresponding to the to-be-retrieved archive, to obtain a comprehensive semantic similarity of the to-be-retrieved archive, thereby sorting the to-be-retrieved archives to determine a semantic retrieval sorting result of the natural language query sentence. The application can improve the accuracy of retrieval result sorting.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, specifically to a semantic retrieval ranking method and related equipment based on archive hierarchy weights. Background Technology

[0002] With the advancement of digital transformation of archives, archives at all levels and enterprises have accumulated a massive amount of electronic archival resources. In order to meet users' needs for querying archival information, existing archival management systems generally adopt semantic retrieval technology based on keyword matching or general deep learning models. This involves comparing the natural language query entered by the user with the archival text content in the system database, and ranking and displaying the search results based on the calculated text similarity scores.

[0003] However, existing general semantic retrieval methods typically treat all the constituent texts of a document (such as the case file title and main content) as a homogeneous whole and perform undifferentiated vectorization calculations. This feature equivalence mechanism caused by data flattening dilutes the core metadata (such as the case file title) with high representational value in the document with massive amounts of text information. As a result, many irrelevant documents that only occasionally mention keywords in the main text receive high similarity scores due to the large amount of text, thus causing the core documents to be submerged in noisy data and resulting in low accuracy in ranking search results. Summary of the Invention

[0004] The embodiments of this application provide a semantic retrieval ranking method and related equipment based on archive hierarchy weights, which aims to improve the accuracy of retrieval result ranking.

[0005] In a first aspect, embodiments of this application provide a semantic retrieval ranking method based on archive hierarchy weights, the semantic retrieval ranking method based on archive hierarchy weights including:

[0006] The system obtains a natural language query statement input by the user and multiple files to be retrieved, wherein each file to be retrieved includes multiple levels of text, including archive level text, directory level text, volume level text, file level text and page level text.

[0007] For each of the aforementioned archives to be retrieved, the hierarchical semantic similarity between the natural language query statement and the archive's archival-level text, catalog-level text, volume-level text, document-level text, and page-level text is determined.

[0008] Obtain multiple hierarchical weights corresponding to the archive to be retrieved, wherein the multiple hierarchical weights include archival level weight, catalog level weight, volume level weight, document level weight, and page level weight;

[0009] According to the multiple hierarchical weights corresponding to the file to be retrieved, the hierarchical semantic similarity of the file to be retrieved is weighted and summed to obtain the comprehensive semantic similarity of the file to be retrieved;

[0010] Based on the comprehensive semantic similarity, the multiple files to be retrieved are sorted to determine the semantic retrieval ranking result of the natural language query statement.

[0011] In the above embodiments, by calculating hierarchical semantic similarity for the text at the archival group, catalog, volume, document, and page levels of the archive to be retrieved, and introducing corresponding weights for each level, this embodiment of the application can take into account the semantic matching differences of natural language query statements at different archival structural levels when calculating the comprehensive semantic similarity. This multi-dimensional weighted summation processing ensures that the final determined semantic retrieval ranking results accurately reflect the substantial relevance between the archive to be retrieved and the query content, thereby effectively improving the accuracy of the retrieval result ranking.

[0012] In one embodiment, determining the hierarchical semantic similarity between the natural language query statement and the archival-level text, catalog-level text, volume-level text, document-level text, and page-level text of the document to be retrieved includes:

[0013] Call a pre-built offline semantic index library, wherein the offline semantic index library pre-stores hierarchical semantic vectors corresponding to multiple hierarchical texts;

[0014] Using a pre-trained language model, a query semantic vector corresponding to the natural language query statement is generated;

[0015] Calculate the cosine similarity value between the query semantic vector and the hierarchical semantic vector of each level of text, and use it as the corresponding hierarchical semantic similarity.

[0016] In the above embodiments, by calling a pre-built offline semantic index library and using a pre-trained language model to generate query semantic vectors, the computational burden of real-time vectorization encoding of massive hierarchical texts during the retrieval stage is avoided. By calculating the cosine similarity between the query semantic vector and the hierarchical semantic vectors, the distance between the natural language query statement and texts at various levels such as archival, volume, and page can be accurately quantified in the multi-dimensional vector space. This ensures that the determined hierarchical semantic similarity can objectively reflect the deep semantic relationships between texts, providing an accurate data foundation for subsequent weighted ranking.

[0017] In one embodiment, the offline semantic index library is constructed through the following steps:

[0018] Parse the hierarchical structure of the archive to be retrieved to determine the parent-child node dependencies among the archival level text, directory level text, volume level text, document level text, and page level text in the archive to be retrieved;

[0019] According to the parent-child node dependency relationship, the text content of the parent node level is accumulated and concatenated to the beginning of the text content of the child node level to generate the fused level text corresponding to each level of text;

[0020] Using the pre-trained language model, a fusion semantic vector corresponding to the fusion level text is generated, and this vector serves as the level semantic vector of the corresponding level text, forming the offline semantic index library.

[0021] In the above embodiments, by parsing the parent-child node dependency relationship and concatenating the accumulated text content of the parent node level to the header of the child node level text, the generated fused hierarchical text can fully inherit and reflect the overall structured context of the archive to be retrieved. Based on this, a pre-trained language model is used to generate fused semantic vectors and form an offline semantic index library, eliminating the semantic ambiguity that may be generated by isolated text fragments at the lower level, and ensuring that the hierarchical semantic vectors of micro-levels such as document and page contain the background features of macro-levels such as archive and volume, thereby enhancing the completeness of semantic representation.

[0022] In one embodiment, obtaining the multiple hierarchical weights corresponding to the file to be retrieved includes:

[0023] Semantic features are extracted from the natural language query statement to obtain the entity type density feature and / or keyword granularity feature of the natural language query statement;

[0024] The entity type density feature and / or the keyword granularity feature are input into a preset query intent recognition model to determine the query intent category of the natural language query statement;

[0025] In the preset intent-weight mapping table, find the target weight allocation strategy that matches the query intent category;

[0026] Based on the target weight allocation strategy, the hierarchical weights corresponding to the archival data at the collection level, catalog level, volume level, document level, and page level are determined respectively.

[0027] In the above embodiments, by extracting entity type density features and keyword granularity features from natural language query statements and using a query intent recognition model to determine the query intent category, the true search focus of the user input can be deeply analyzed. A matching target weight allocation strategy is searched in a preset intent-weight mapping table, achieving dynamic adaptation of weight values ​​for different levels such as archives, volumes, documents, and pages. This ensures that the weight ratio during weighted summation processing remains consistent with the user's current intent, thereby improving the relevance of the search ranking results.

[0028] In one embodiment, after determining the semantic retrieval ranking result of the natural language query statement based on the ranking result, the method further includes:

[0029] Obtain user click behavior data on the semantic search ranking results;

[0030] Based on the file hierarchy information corresponding to the click behavior data, determine the user's preference characteristics for different file hierarchy information;

[0031] Based on the preference features, determine the weight adjustment value for each level weight;

[0032] The target weight allocation strategy in the intent-weight mapping table is updated using the weight correction value.

[0033] In the above embodiments, by acquiring user click behavior data and combining it with archive hierarchy information to determine preference features, the distribution of user attention to different levels of content can be quantified using actual interaction records. Based on preference features, weight correction values ​​are determined and the target weight allocation strategy is updated, constructing an adaptive calibration closed loop for weight configuration. This allows the data in the intent-weight mapping table to automatically optimize as actual business scenarios evolve, ensuring that the hierarchical weight settings continuously approximate users' actual search expectations and guaranteeing the long-term effectiveness of the ranking logic.

[0034] In one embodiment, finding a target weight allocation strategy that matches the query intent category in a preset intent-weight mapping table includes:

[0035] If the query intent category is a macro navigation intent, the first weight allocation strategy in the intent-weight mapping table is used as the target weight allocation strategy, wherein the first weight allocation strategy represents that the level weight corresponding to the archival level text and the level weight corresponding to the volume level text are both greater than the level weight corresponding to the document level text and the level weight corresponding to the page level text.

[0036] If the query intent category is a micro-detail intent, the second weight allocation strategy in the intent-weight mapping table is used as the target weight allocation strategy, wherein the second weight allocation strategy indicates that the level weight corresponding to the archival level text and the level weight corresponding to the volume level text are both less than the level weight corresponding to the document level text and the level weight corresponding to the page level text.

[0037] In the above embodiments, by distinguishing between macro-level navigation intent and micro-level detail intent, and respectively matching a first weight allocation strategy that emphasizes archival records and volumes or a second weight allocation strategy that emphasizes documents and pages, the system can accurately handle two different retrieval scenarios: general category positioning and specific content searching. This processing ensures that top-level archival information is prioritized for macro-level queries, while highlighting the details of the underlying text for micro-level queries, thus effectively solving the problem of sorting disorder caused by a single weight system being unable to accommodate different granularity retrieval needs.

[0038] In one embodiment, the step of extracting semantic features from the natural language query statement to obtain the entity type density feature and / or keyword granularity feature of the natural language query statement includes:

[0039] Named entity recognition is performed on the natural language query statement to extract at least one of the following entities: organization name entity, year entity, and business terms entity;

[0040] The entity type density feature is obtained by determining the ratio between the number of characters occupied by the organization name entity and / or the year entity and the total number of characters in the natural language query statement;

[0041] And / or, obtain the inverse document frequency value of the business term entity in the offline semantic index library, determine the semantic specificity of the business term entity based on the inverse document frequency value, and use it as the keyword granular feature.

[0042] In the above embodiments, entity type density features are obtained by performing named entity recognition and calculating the ratio of the number of characters occupied by organization names or year entities, and semantic specificity of business clause entities is determined based on inverse document frequency values ​​as keyword granularity features, thus transforming abstract natural language query statements into specific statistical indicators. This provides the query intent recognition model with quantitative input data with clear physical meaning, ensuring that the intent classification results originate from objective linguistic rules, thereby improving the robustness and reliability of query intent category determination.

[0043] Secondly, embodiments of this application provide a semantic retrieval ranking system based on archive hierarchy weights, wherein the semantic retrieval ranking system based on archive hierarchy weights runs on an electronic device and is used to execute the semantic retrieval ranking method based on archive hierarchy weights as described in any of the preceding claims.

[0044] Thirdly, embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the semantic retrieval ranking method based on archive hierarchy weights as described in any of the preceding claims.

[0045] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program configured to be executed by a processor to implement the semantic retrieval ranking method based on archive hierarchy weights as described in any of the preceding claims. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic flowchart of an embodiment of the semantic retrieval ranking method based on archive hierarchy weight provided in this application;

[0048] Figure 2 This is a schematic diagram of an embodiment of the electronic device provided in this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. In addition, in the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0050] In a first aspect, embodiments of this application provide a semantic retrieval ranking method based on archive hierarchy weights, applied to electronic devices, with the executing entity being a semantic retrieval ranking system based on archive hierarchy weights (hereinafter referred to as the "system"), which runs on electronic devices.

[0051] Specifically, refer to Figure 1 , a semantic retrieval sorting method based on file - level weights may include:

[0052] S101. Obtain a natural - language query statement input by a user, and obtain a plurality of files to be retrieved. Each file to be retrieved includes multiple hierarchical texts, and the multiple hierarchical texts include fonds - level texts, directory - level texts, volume - level texts, piece - level texts, and page - level texts.

[0053] In an embodiment of the present application, a natural - language query statement refers to a text string in which a user expresses a retrieval intention in unstructured daily language, which may contain keywords, interrogative words, or descriptive phrases. A file to be retrieved refers to a digital file resource stored in the database of a file - management system. Fonds - level texts, directory - level texts, volume - level texts, piece - level texts, and page - level texts refer to metadata or content texts corresponding to the hierarchical division of a file entity according to file - industry standards. Among them, fonds - level texts represent the highest management level of a file, usually corresponding to the name information of a specific filing unit or individual; directory - level texts represent classification - unit information under a fonds; volume - level texts represent the collection information of a group of closely related documents; piece - level texts represent the title or responsible - person information of a single document; page - level texts represent the text content information of a specific page of a document.

[0054] In some embodiments of the present application, after obtaining a natural - language query statement, the statement will be pre - processed. The pre - processing process mainly includes removing stop words (such as function words like "of", "about", etc.) and removing irrelevant punctuation marks to reduce the impact of non - key characters on semantic - feature extraction. If the application scenario involves multi - round continuous queries, the obtaining step further includes extracting historical query context information, and using natural - language - processing techniques to perform anaphora resolution or ellipsis completion on the currently input natural - language query statement, so as to generate an enhanced query statement containing a complete retrieval intention. For example, when a user inputs "XX Company" and "contracts in 2020" successively, the system will combine the two to generate "XX Company contracts in 2020" as the final natural - language query statement.

[0055] In some embodiments of the present application, the hierarchical texts of each file to be retrieved can be pre - extracted and stored for subsequent use. Specifically, for digital scanned documents, text can be extracted through Optical Character Recognition (OCR) technology, and the extraction result can be cleaned to remove garbled characters and redundant spaces, and the hierarchical texts of the file to be retrieved are obtained and stored. In addition, term standardization processing can also be performed on texts with specific formats such as fonds - level texts to ensure data consistency.

[0056] S102. For each archive to be retrieved, determine the hierarchical semantic similarity between the natural language query statement and the archival-level text, catalog-level text, volume-level text, document-level text, and page-level text of the archive to be retrieved.

[0057] In the embodiments of this application, hierarchical semantic similarity refers to a numerical indicator used to quantify the degree of semantic association between a query statement and a specific level of text in an archive.

[0058] In some embodiments of this application, the process of determining hierarchical semantic similarity includes: using a pre-trained language model (PLM), such as a Bidirectional Encoder Representations from Transformers (BERT) or a Robustly Optimized BERT Pretraining Approach (RoBERTa), to map the natural language query statement and the archival group-level, directory-level, volume-level, document-level, and page-level texts of the archive to be retrieved into high-dimensional dense semantic vectors, respectively. Subsequently, a cosine similarity algorithm is used to calculate the cosine value of the angle between the vector generated from the natural language query statement and the vectors generated from each level of text, and this cosine value is used as the corresponding hierarchical semantic similarity. This value is typically normalized to the range of 0 to 1.

[0059] In some embodiments of this application, considering that general category level text is usually shorter (mainly names) while page level text is longer (mainly body text), differentiated encoding strategies can be adopted for different levels when generating hierarchical semantic vectors. For example, for short text levels, specific classification markers (such as [CLS] bits) vectors output by the model are directly used as representations; for long text levels, a sliding window strategy is used to segment the text, encode each segment separately, and then perform mean pooling to ensure that the semantic features of long documents are not truncated, thereby ensuring the accuracy of hierarchical semantic similarity calculation.

[0060] S103. Obtain the weights of multiple levels corresponding to the archive to be searched, including the weights of the archival group, the weights of the catalog, the weights of the volume, the weights of the document, and the weights of the page.

[0061] In this embodiment, the hierarchical weight refers to a coefficient that assigns importance to different levels of text in the overall semantic expression of the archive. The archival level weight, catalog level weight, volume level weight, document level weight, and page level weight correspond to the above five levels, and the sum of the values ​​of these five weights is usually set to 1.

[0062] In some embodiments of this application, the determination of hierarchical weights is based on the Analytic Hierarchy Process (AHP) or expert scoring, with an emphasis on increasing the weight of the core metadata layer. For example, since archival retrieval typically focuses primarily on fonds ownership and file titles, the weights of fonds and volumes are set higher than those of pages. For instance, the fonds weight is set to 0.3, the volume weight to 0.3, the file weight to 0.2, the catalog weight to 0.1, and the page weight to 0.1.

[0063] In some embodiments of this application, the hierarchical weights are not static but dynamically optimized through an offline evaluation process. The system collects historical retrieval samples, calculates the retrieval precision and recall under different weight configurations, and fine-tunes the specific values ​​of each hierarchical weight through gradient descent or grid search algorithms to ensure that the stored hierarchical weights can best adapt to the current archive structure features.

[0064] S104. According to the weights of multiple levels corresponding to the archive to be retrieved, perform weighted summation of the hierarchical semantic similarity of the archive to be retrieved to obtain the comprehensive semantic similarity of the archive to be retrieved.

[0065] In this embodiment of the application, the comprehensive semantic similarity refers to the comprehensive score that represents the overall matching degree between the file to be retrieved and the user's query intent.

[0066] In some embodiments of this application, the specific calculation logic of the weighted summation process is as follows: multiply the weight of the archival level by the hierarchical semantic similarity corresponding to the archival level text to obtain a first product; multiply the weight of the directory level by the hierarchical semantic similarity corresponding to the directory level text to obtain a second product; multiply the weight of the volume level by the hierarchical semantic similarity corresponding to the volume level text to obtain a third product; multiply the weight of the document level by the hierarchical semantic similarity corresponding to the document level text to obtain a fourth product; multiply the weight of the page level by the hierarchical semantic similarity corresponding to the page level text to obtain a fifth product; finally, add the first to fifth products together, and the sum is the comprehensive semantic similarity of the archive to be retrieved. Through this calculation method, even if the query term is mentioned multiple times in the page level text (body text) of a certain archive, if its archival level text or volume level text does not match the query intent, its final comprehensive semantic similarity will be reduced by the weight coefficient, thereby effectively reducing the impact of non-core level text on the comprehensive semantic similarity calculation result.

[0067] S105. Sort the multiple files to be retrieved according to the comprehensive semantic similarity, so as to determine the semantic retrieval ranking result of the natural language query statement based on the ranking result.

[0068] In this embodiment of the application, the semantic retrieval ranking result refers to the ordered list of files finally presented to the user after semantic matching and rule sorting.

[0069] In some embodiments of this application, in step S105, the system sorts all the calculated comprehensive semantic similarity scores of the files to be retrieved in descending order. The system can preset a similarity threshold (e.g., 0.5) and only retain files whose comprehensive semantic similarity is greater than the threshold. The final sorting results will be displayed in descending order of comprehensive semantic similarity, and the collection-level text (e.g., collection name) and volume-level text (e.g., volume name) of the file will be displayed simultaneously on the display interface to help users quickly confirm the accuracy of the search.

[0070] As can be seen, this application's embodiments, by constructing a five-dimensional hierarchical weight system adapted to archival industry standards, refine the granularity of semantic matching to specific levels such as archival collections, catalogs, volumes, documents, and pages, and calculate the comprehensive similarity by combining differentiated hierarchical weights. This solution effectively solves the problem in general retrieval where the weight of core metadata information is too low due to the large amount of text data, achieving accurate ranking based on archival professional hierarchical logic, and improving the precision of archival resource retrieval in complex semantic scenarios and the efficiency of users obtaining core archives.

[0071] After detailing the overall framework and basic operational flow of the semantic retrieval ranking method based on archive hierarchy weights in the above embodiments, in order to further improve the real-time response capability and calculation accuracy of the system when processing large-scale archive data, it is necessary to specify the technical limitations of the core "hierarchical semantic similarity" calculation method. Compared with online real-time encoding, the combination of offline indexing and online comparison is more efficient in engineering implementation. Based on this, some embodiments of this application will describe in detail the specific implementation process of determining hierarchical semantic similarity in step S102. Specifically, the hierarchical semantic similarity between the natural language query statement and the archival data at the archival data level (archive collection level, catalog level, volume level, document level, and page level) is determined, including:

[0072] S201. Call the pre-built offline semantic index library, which pre-stores hierarchical semantic vectors corresponding to multiple levels of text.

[0073] In this embodiment, the offline semantic index library refers to a dedicated database or data structure that is pre-computed and persistently stored, used to store high-dimensional numerical features generated after the archival data has been vectorized and encoded. The hierarchical semantic vector refers to a fixed-dimensional array obtained by mapping archival hierarchical text, directory hierarchical text, volume hierarchical text, document hierarchical text, and page hierarchical text to a vector space using a deep learning model; it can preserve the semantic information of the original text.

[0074] In some embodiments of this application, the construction process of the offline semantic index library is independent of the user's real-time search requests. The system periodically extracts all text content, including archival level text, catalog level text, volume level text, document level text, and page level text, for each document to be searched, either in the background or when the archival data is added to the database. This content is then batch-converted into corresponding hierarchical semantic vectors using a pre-defined encoding model. These hierarchical semantic vectors are typically stored in the database using the document's unique identifier as the index key and a structure containing five independent vectors (corresponding to the five levels) as the index value. Using an offline construction method avoids the enormous computational overhead of calculating massive amounts of archival text vectors in real-time when a user initiates a query, thus ensuring the system's response speed.

[0075] S202. Using a pre-trained language model, generate the query semantic vector corresponding to the natural language query statement.

[0076] In this embodiment, the pre-trained language model refers to a deep neural network model, such as the BERT model, that has undergone unsupervised learning on a large-scale general corpus and possesses general language understanding capabilities. The query semantic vector refers to the feature vector output after the natural language query statement is encoded by this model.

[0077] In some embodiments of this application, the system first performs tokenization on the natural language query statement, converting it into a word embedding sequence acceptable to the model. Subsequently, the processed sequence is input into a pre-trained language model for forward propagation inference. To ensure semantic space consistency, the pre-trained language model used here must maintain a completely consistent parameter architecture with the model used to build the offline semantic index library in step S201. The system extracts all output vectors from the last hidden layer of the model and performs average pooling to obtain a query semantic vector that can represent the core intent of the natural language query statement.

[0078] S203. Calculate the cosine similarity between the query semantic vector and the hierarchical semantic vector of each level of text, and use it as the corresponding hierarchical semantic similarity.

[0079] In the embodiments of this application, the cosine similarity value refers to the cosine value of the angle between two vectors in multidimensional space, which is used to measure the degree of closeness of two vectors in direction, and is not affected by the vector magnitude.

[0080] In some embodiments of this application, for each document to be retrieved, the system reads the hierarchical semantic vectors of the following text from an offline semantic index: the archival level text, the directory level text, the volume level text, the document level text, and the page level text. Subsequently, the system performs vector operations, calculating the dot product of the query semantic vector with each of the five hierarchical semantic vectors, and dividing by the product of the magnitudes of the two vectors, thus obtaining five independent cosine similarity values. These five values ​​correspond to the hierarchical semantic similarity between the natural language query and the document to be retrieved at five different levels. Through this step, the abstract text matching problem is transformed into a precise mathematical vector operation problem.

[0081] As can be seen, the embodiments of this application decouple the extraction of archival text features from the online retrieval process by pre-constructing an offline semantic index library, significantly reducing the computational load in the real-time retrieval stage. Simultaneously, by utilizing a general pre-trained language model to map natural language query statements and archival text at various levels to a unified semantic vector space, and employing cosine similarity for measurement, the accuracy of semantic matching and computational efficiency are ensured, enabling the system to maintain millisecond-level retrieval response capabilities even when faced with massive amounts of archival data.

[0082] The above embodiments clarify the specific path for vectorized comparison using an offline semantic index library and a pre-trained language model. However, the quality of the offline semantic index library directly determines the upper limit of retrieval. In archival management practice, low-level text (such as "item" or "page") often suffers from semantic ambiguity or vagueness due to its separation from the upper-level archival context. Directly performing isolated vectorized encoding will result in incomplete semantic expression of the index. To address this data-level deficiency and ensure that each vector stored in the library can carry complete contextual information, the following embodiments of this application will focus on describing the construction details of the offline semantic index library and its unique text fusion mechanism. Specifically, the offline semantic index library is constructed through the following steps:

[0083] S301. Parse the hierarchical structure of the corresponding archive to be retrieved to determine the parent-child node dependencies between the archival level text, catalog level text, volume level text, document level text, and page level text in the archive to be retrieved.

[0084] In this embodiment, the parent-child node dependency relationship refers to the inclusion and attribution logic of a higher-level category to a lower-level category in the file management architecture. For example, a volume-level text is a child node of a directory-level text, and at the same time, it is the parent node of an item-level text.

[0085] In some embodiments of this application, in step S301, the system reads the metadata configuration file or database association table of the archive to be retrieved and constructs an archive multi-branch tree with the archival level text as the root node and the page level text as the leaf nodes. The system traverses the archive multi-branch tree, marking the direct predecessor node (parent node) and indirect predecessor node (ancestor node) of each non-root node level text, thereby clarifying the specific location path of each independent text block in the overall archival architecture and establishing a complete parent-child node dependency mapping table.

[0086] S302. According to the parent-child node dependency relationship, accumulate and concatenate the text content of the parent node level to the header of the text content of the child node level to generate the fused level text corresponding to each level of text.

[0087] In this embodiment, the fused hierarchical text refers to a text sequence after context enhancement processing, which includes not only the text content of the current level itself, but also the text content of all or part of its parent nodes along its path. Cumulative concatenation refers to combining text using string concatenation operations in descending order of hierarchy.

[0088] In some embodiments of this application, in step S302, to address the problem of limited semantic information and lack of contextual constraints in low-level text (such as a simple document name "Meeting Minutes" or page text), a full-path inheritance strategy is adopted to generate fused hierarchical text. Specifically, for a target child node hierarchical text (e.g., document hierarchical text), the system traces back its parent-child node dependencies and extracts all parent node hierarchical texts along the path from the root node (archive) to the target node. Subsequently, the system uses specific delimiters (such as spaces or special characters) to sequentially concatenate the archive hierarchical text, directory hierarchical text, and volume hierarchical text at the beginning of the document hierarchical text. For example, the generated fused hierarchical text format is: "[archive name] [directory name] [volume name] [document name]". For page hierarchical text, the main text content is further concatenated on top of the document hierarchical text. If the cumulative concatenated text length exceeds the maximum input limit of the pre-trained language model, the system prioritizes retaining the current hierarchical text and the nearest neighbor parent node text, and truncates the more distant ancestor node text.

[0089] S303. Using a pre-trained language model, generate fusion semantic vectors corresponding to the fusion level texts, and use them as the hierarchical semantic vectors of the corresponding level texts to form an offline semantic index library.

[0090] In this embodiment of the application, the fused semantic vector refers to the numerical feature that will be operated on in a high-dimensional space, which contains the background context semantics of the archive.

[0091] In some embodiments of this application, in step S303, the fused hierarchical text generated in step S302 is input into a pre-trained language model (such as the BERT model). Since the input text already contains background information of the parent nodes, when the model performs self-attention mechanism calculation, it can perform domain-specific semantic disambiguation on the child node text (such as "contract") based on the parent node text (such as "XX project"), thereby generating a fused semantic vector with strong context awareness. The system uses this fused semantic vector as the final hierarchical semantic vector of the hierarchical text and stores it in an offline semantic index library. In subsequent retrieval and comparison, the comprehensive semantics of "collection + catalog + volume + item" are actually compared with the query statement, rather than just comparing the isolated "item" text.

[0092] As can be seen, this application's embodiments introduce hierarchical contextual information of archives during the vectorization encoding stage by employing a text accumulation and concatenation strategy based on parent-child node dependencies. This scheme effectively overcomes the problem of semantic ambiguity caused by the lack of background description in low-level archive units (such as single items or isolated pages with generic filenames) during archive retrieval, ensuring that the generated index vector has semantic constraints at the archival and file dimensions, thereby improving the retrieval discrimination and accuracy of archive resources under specific archival backgrounds.

[0093] The aforementioned embodiments solved the problems of "semantic representation" and "index construction" of archival texts, building an accurate and structured data foundation. However, at the logical level of retrieval and ranking, a single, fixed weight system is difficult to adapt to the diverse query scenarios of users. In order for the system to have adaptive judgment capabilities, that is, to dynamically adjust the scoring criteria based on whether the user is "searching for the entire archive" or "searching for specific clauses," a deep understanding mechanism of the user's natural language query statements needs to be introduced. To this end, some embodiments of this application will return to the weight acquisition stage, illustrating how to drive the dynamic allocation of hierarchical weights by recognizing query intent. Specifically, multiple hierarchical weights corresponding to the archive to be retrieved are acquired, including:

[0094] S401. Extract semantic features from natural language query statements to obtain entity type density features and / or keyword granularity features of natural language query statements.

[0095] In this embodiment, the entity type density feature refers to the proportion of named entities with specific meanings (such as personal names, place names, organization names, time adverbs, etc.) in a natural language query statement to the total length of the statement. This feature is used to measure whether a user query tends to use specific attribute information for precise searching. The keyword granularity feature refers to the generality or specificity of words in a natural language query statement, which is usually characterized by the Term Frequency-Inverse Document Frequency (TF-IDF) value. High-frequency general words (such as "notice" or "letter") correspond to coarser granularity, while low-frequency specialized words (such as specific project codes) correspond to finer granularity.

[0096] In some embodiments of this application, in step S401, the system calls a Named Entity Recognition (NER) tool to analyze the natural language query statement, counts the number of entities such as time, location, people, and archival records contained within it, and calculates the ratio of this number to the total number of words in the statement to obtain the entity type density feature. Simultaneously, the system combines a pre-built archival professional dictionary to calculate the average information entropy or inverse document frequency of the content words in the query statement to generate keyword granularity features. For example, the query statement "2023 Financial Audit Report" contains a clear time and a specific business type, and its entity type density feature and keyword granularity feature are both high.

[0097] S402. Input the entity type density features and / or keyword granularity features into the preset query intent recognition model to determine the query intent category of the natural language query statement.

[0098] In this embodiment, the query intent recognition model refers to a trained machine learning classifier, such as a Support Vector Machine (SVM) or a Multilayer Perceptron (MLP). This model is used to establish a mapping relationship from text features to the user's potential query intent categories. Query intent categories include, but are not limited to, "macro-navigation intent" (preferring to search for case files or document titles) and "micro-detail intent" (preferring to search for specific paragraphs in the main text).

[0099] In some embodiments of this application, in step S402, the system inputs a feature vector composed of the features extracted in step S401 into the model. If the model identifies that the input vector has high entity type density features (e.g., containing explicit years and organization names), the query intent category is determined to be "macro navigation intent," meaning that the user may be looking for a specific volume or document; if the input vector shows low keyword granularity features and sparse entities (e.g., querying "regulations on fire safety"), it is determined to be "micro detail intent," meaning that the user is more concerned with the specific details involved in the text.

[0100] In some embodiments of this application, the construction and offline training process of the query intent recognition model specifically includes the following data processing logic:

[0101] Sample set construction: The system extracts no less than 100,000 historical query statements from the historical retrieval logs. For each historical query statement, the same feature extraction algorithm as in step S401 is used to calculate a two-dimensional feature vector composed of [entity type density feature value, keyword granularity feature value], which is used as the training input data (X). For example, the feature vector is [0.85, 0.20].

[0102] Labeling: Historical query statements are labeled as either "macro-level navigation intent" (label value set to 0) or "micro-level detail intent" (label value set to 1) using manual labeling or semi-supervised clustering. The labeling rules are as follows: if the query statement explicitly contains a domain name, file number, or specific document number format, it is labeled as 0; if the query statement is a pure natural language description without specific entity constraints, it is labeled as 1. This label serves as the training target data (Y).

[0103] Model training: Logistic Regression or Support Vector Machine (SVM) is selected as the base model. The sample set is divided into training and validation sets in an 8:2 ratio.

[0104] Kernel function selection: If SVM is used, the Radial Basis Function (RBF) is selected to map the two-dimensional feature vector to a high-dimensional space to solve the problem of nonlinear separability of features in low-dimensional space.

[0105] Convergence criteria: Set the loss function to cross-entropy loss and use stochastic gradient descent (SGD) to iteratively update the loss function until the classification accuracy on the validation set exceeds 95% or the loss function value decreases by less than 1e-5. Output the trained model parameter file.

[0106] Through the training process disclosed above, those skilled in the art can reproduce a query intent recognition model with the same function, ensuring the determinism of the mapping relationship between input features and output categories.

[0107] S403. In the preset intent-weight mapping table, find the target weight allocation strategy that matches the query intent category.

[0108] In this embodiment, the intent-weight mapping table refers to a data table that stores the correspondence between different query intent categories and the optimal hierarchical weight combination. The target weight allocation strategy refers to a set of predefined numerical values ​​for the archive level weight, directory level weight, volume level weight, file level weight, and page level weight optimized for a specific intent.

[0109] In some embodiments of this application, in step S403, if the query intent category is "macro navigation intent", the strategy found by the system in the table will focus on high-level information, such as setting the weight of the archive level and the volume level to a higher value (e.g., 0.35 each), while reducing the weight of the page level (e.g., 0.05); if the query intent category is "micro detail intent", the strategy found will increase the weight of the page level (e.g., increase it to 0.6) to ensure that files not reflected in the title but containing relevant descriptions in the body text can be retrieved.

[0110] S404. Based on the target weight allocation strategy, determine the corresponding hierarchical weights for the archival data at the archival data level, including the archival data level, the catalog data level, the volume data level, the document data level, and the page data level.

[0111] In this embodiment of the application, the system directly parses the five numerical parameters contained in the target weight allocation strategy and assigns them to the five hierarchical weight variables of the file to be retrieved.

[0112] In some embodiments of this application, the system also supports fine-tuning and smoothing of weights. If the classification confidence of the model in step S402 is between two categories, the system can perform linear interpolation on the weight strategies corresponding to the two categories to generate a set of transitional hierarchical weights, thereby preventing drastic fluctuations in search results caused by abrupt changes in the intention classification boundary.

[0113] As can be seen, this embodiment extracts the entity density and granularity features of natural language query statements and uses an intent recognition model to intelligently determine the user's search motivation. Based on this, the system dynamically adjusts the weight allocation strategy for different levels such as archival data, volume, and page. For precise search intents, it emphasizes title matching, while for generalized search intents, it emphasizes text matching. This scheme achieves adaptive perception of user intent by the retrieval model, ensuring that the weight system accurately reflects the core levels of user interest in different query scenarios, further improving the intelligence and accuracy of search ranking.

[0114] While dynamic weight allocation strategies based on intent recognition endow the system with a certain degree of adaptability, the initial "intent-weight" mapping relationship is usually based on expert experience or prior rules, which may lead to deviations that do not fully align with actual user psychology. To enable the system to continuously evolve and form various closed loops of "retrieval-use-optimization," user behavior data needs to be introduced as a basis for post-hoc correction. Therefore, some embodiments of this application will further illustrate how to utilize user click interaction behavior to reverse-correct and iteratively update the weight allocation strategy. Specifically, after determining the semantic retrieval ranking result of the natural language query statement based on the ranking result, the method further includes:

[0115] S501. Obtain user click behavior data on semantic search ranking results.

[0116] In this embodiment, click behavior data refers to log information recording user interactions with the displayed search results list on the client interface. This data includes not only the clicked profile identifier (ID), but also the timestamp of the click, the dwell time on the target page, and the specific location of the clicked area.

[0117] In some embodiments of this application, in step S501, the system collects user interaction events in real time using front-end tracking technology. Specifically, to differentiate the levels of user focus, the system records whether the user clicks on the title area (usually corresponding to volume-level or document-level text) or the summary preview area (usually corresponding to page-level text) in the results overview. Furthermore, if the system interface explicitly highlights "Hit Records" or "Hit Catalog," the user's click on that specific tag is also considered key click behavior data. The system filters out erroneous clicks with a dwell time below a preset threshold (e.g., 3 seconds), retaining only valid click data for subsequent analysis.

[0118] S502. Based on the file hierarchy information corresponding to the click behavior data, determine the user's preference characteristics for different file hierarchy information.

[0119] In this embodiment, the archive hierarchy information refers to the specific attributes of the archival collections, catalogues, volumes, documents, or pages that are primarily displayed or hit in the search results that trigger user clicks. Preference features refer to the numerical distribution quantified through statistical methods, reflecting the degree of user dependence on a certain level of information in a specific query scenario.

[0120] In some embodiments of this application, in step S502, the system uses attribution analysis to determine preference features. For each valid click, the system backtracks the score composition of the file during the sorting calculation. If the overall semantic similarity of a clicked file mainly comes from the product of the archive level weight and the archive level semantic similarity, then the click is determined to be positive feedback to the archive level information; if the main contribution comes from the page level, then it is determined to be positive feedback to the page level information. The system will accumulate statistics on all click behaviors under the same query intent category within a statistical period (e.g., one week), calculate the frequency ratio of positive feedback at each level, and generate a set of normalized values ​​(e.g., volume level preference 0.5, document level preference 0.3, page level preference 0.2) as the user preference features under that intent.

[0121] In the embodiments of this application, the specific calculation logic for determining the user's preference features for information at different archive levels can employ a weighted score contribution algorithm, as follows:

[0122] Obtaining Sub-item Scores: When a user clicks on the i-th file in the sorted list, the system reads the sub-item scores for each level calculated during the sorting process. Let the weight of the archival archive level be W1, and the semantic similarity of the archival archive level be S1; the weight of the page level be W5, and the semantic similarity of the page level be S5 (and so on for other levels).

[0123] Contribution Calculation: The system calculates the weighted score of each level relative to the total ranking score of the file, which is used as the contribution value of this click to each level. Specifically, the contribution C of the k-th level (k∈{archives, catalogs, volumes, documents, pages}) is... k The calculation formula is:

[0124]

[0125] For example, in a certain click, the weighted score for the entire record level is 0.3, the weighted score for the page level is 0.7, and the total score is 1.0. Therefore, this click contributes 0.3 to the preference for the entire record level and 0.7 to the preference for the page level.

[0126] Aggregate statistics: Within the statistical period T, the system accumulates the contribution of all valid clicks under the same query intent category and calculates the average value to obtain a normalized preference feature vector P=[Parchment, PDirectory, PVolume, PItem, PPage]. The specific formula is:

[0127]

[0128] Where M represents the total number of valid clicks within the statistical period, and C... k,m This represents the contribution of the k-th level in the m-th click.

[0129] Through the processing logic defined by the mathematical formulas described above, the system can quantify the underlying motivations behind user click behavior, avoiding the bias caused by simple counting methods (i.e. avoiding the mistaken assumption that all clicks are due to text matching), thus providing clear data support for the calculation of preference features.

[0130] S503. Based on preference features, determine the weight adjustment value for each level of weight.

[0131] In this embodiment of the application, the weight correction value refers to the incremental or decremental value used to fine-tune the set value stored in the current intent-weight mapping table.

[0132] In some embodiments of this application, in step S503, the system compares the calculated preference features with the original weights in the currently effective target weight allocation strategy. If the preference feature value of a certain level (e.g., a volume level) is higher than its current set weight, it indicates that the current weight underestimates the importance of that level, and the system generates a positive weight correction value; otherwise, a negative weight correction value is generated. To maintain system stability and avoid significant fluctuations in weight values ​​due to individual abnormal data, the system can introduce a decay coefficient or learning rate. The calculation formula can be expressed as: Weight correction value = Learning rate × (Preference feature value - Current weight value). Simultaneously, the system ensures that the sum of the correction values ​​for all level weights is zero to maintain the characteristic of total weight normalization.

[0133] S504. Update the target weight allocation strategy in the intent-weight mapping table using the weight correction value.

[0134] In this embodiment of the application, updating refers to applying the calculated weight correction value to the configuration parameters stored in the database, thereby optimizing the system's retrieval logic.

[0135] In some embodiments of this application, in step S504, the system adds the original value of each level's weight to the corresponding weight correction value to obtain the updated level weight, and uses this new value to overwrite the record corresponding to the query intent category in the intent-weight mapping table. This process can be updated online in real time, or it can be executed in batches according to a scheduled task (e.g., during system idle time). Through this step, when a user subsequently initiates a search request belonging to the same query intent category, the system will call the updated target weight allocation strategy for sorting calculation.

[0136] As can be seen, this application embodiment constructs an online learning mechanism based on implicit user feedback. By continuously collecting and analyzing users' actual click behavior, the system can associate the hierarchical attributes of the retrieved archives with the user's true concerns and automatically calculate weight correction values ​​to iteratively optimize the weight configuration strategy. This adaptive closed-loop adjustment ensures that the retrieval ranking algorithm can continuously iterate and optimize as user habits change, so that the weight allocation of archival documents, catalogs, volumes, documents, and pages always aligns with actual business needs, thereby continuously providing more accurate retrieval results that meet user expectations.

[0137] After perfecting the dynamic acquisition and self-correction mechanism of weights, to make the technical solution more complete and specific, the target weight allocation strategy can be concretely implemented in various scenarios. In the field of document retrieval, the most typical differentiation of user needs is the unity of opposites between "macro-level search" and "micro-level search." To clarify the specific weight control logic of the system when facing these two typical intents, the following embodiments of this application will define in detail the differentiated weight configuration principles that the system should adopt under macro-level navigation intents and micro-level detail intents. Specifically, in the preset intent-weight mapping table, the target weight allocation strategy that matches the query intent category is searched, including:

[0138] S601. If the query intent category is macro navigation intent, the first weight allocation strategy in the intent-weight mapping table is used as the target weight allocation strategy. The first weight allocation strategy indicates that the hierarchical weight corresponding to the archival level text and the hierarchical weight corresponding to the volume level text are both greater than the hierarchical weight corresponding to the document level text and the hierarchical weight corresponding to the page level text.

[0139] In this embodiment, macro-navigation intent refers to a user's search intention to locate a specific collection of archives, case files, or management categories, rather than searching for a specific sentence in the main text. Such intent typically includes general terms such as year, institution name, and document type, for example, "document archives of a certain bureau in 2020." The first weight allocation strategy refers to a set of weight parameters specifically designed to meet macro-search needs. This strategy establishes a scoring principle that increases the weight of metadata and decreases the weight of main text data.

[0140] In some embodiments of this application, in step S601, when the intent recognition model determines the intent to be macroscopic navigation, the system reads the first weight allocation strategy from the intent-weight mapping table. The specific numerical settings in this strategy must strictly satisfy the condition that the weights of archives and volumes are higher than the weights of documents and pages. For example, the weight for the archive level is set to 0.35, the weight for the volume level is set to 0.35, the weight for the directory level is set to 0.15, while the weight for the document level is only 0.10, and the weight for the page level is 0.05.

[0141] By implementing this step, the system significantly enhances the influence of archival-level and volume-level texts on the final ranking when calculating comprehensive semantic similarity. If a document only sporadically mentions the query term in the page-level text (body text) but does not match the archival group or volume name, its score will be significantly suppressed. This processing method reuses the retrieval logic of prioritizing matching archival group numbers and file titles in archival management, enabling rapid filtering of target archive boxes from massive databases, avoiding the influence of irrelevant data caused by full-text indexing, and ensuring high accuracy of the returned results in macro-classification.

[0142] S602. If the query intent category is micro-detail intent, the second weight allocation strategy in the intent-weight mapping table is used as the target weight allocation strategy. The second weight allocation strategy indicates that the hierarchical weights corresponding to the archival level text and the volume level text are both less than the hierarchical weights corresponding to the document level text and the page level text.

[0143] In this application embodiment, micro-level detail intent refers to the user's search mentality of needing to find specific facts, data, specific clauses, or names hidden within the archive file. Such intent typically includes specific long-tail keywords or specific proper nouns, such as "a certain agreement's breach of contract clause" or "a specific person's date of birth." The second weighting strategy refers to weight parameters specifically set to meet the needs of in-depth content mining. This strategy establishes a scoring principle of increasing the weight of textual details and decreasing the weight of top-level categories.

[0144] In some embodiments of this application, in step S602, when the system recognizes a micro-level detail intent, a second weight allocation strategy is invoked. In this strategy, since the information the user is looking for typically does not appear directly in the record book name or volume name, the system increases the weight percentage of the lowest-level text. For example, the record book level weight is set to 0.05, the volume level weight to 0.10, the directory level weight to 0.05, while the document level weight is increased to 0.30, and the page level weight is increased to 0.50.

[0145] By implementing this step, even if the full name or volume name is completely unrelated to the natural language query (e.g., the query term is in a file named "Other"), the overall semantic similarity of the file to be retrieved will still be high as long as there is highly semantically matching content in its page-level text (body text). This processing method overcomes the limitations of catalog retrieval, allowing users to effectively capture fine-grained information contained in the document body, thus improving the retrieval recall rate for specific fact searches.

[0146] As can be seen, this application's embodiments construct differentiated bimodal weight allocation logic for two distinct scenarios in document retrieval: macro-level category positioning and micro-level content searching. By dynamically switching between the first and second weight allocation strategies, the system can flexibly adapt to the user's retrieval granularity requirements. This solution prevents information overload during generalized queries and avoids information omission during precise queries, effectively resolving the contradiction that a single static weight cannot simultaneously address different levels of retrieval needs.

[0147] The accurate execution of the aforementioned differentiated weighting strategy relies on the system's ability to precisely classify users' natural language queries into macro-level or micro-level intents. This requires the system to go beyond shallow text matching and delve into quantitative analysis at the syntactic structure and statistical feature levels. To reveal the feature engineering details behind the intent recognition model, the following embodiments of this application will specifically describe how to extract deep features such as entity type density and keyword granularity from query statements to provide a calculable mathematical basis for intent category determination. Specifically, semantic feature extraction is performed on natural language queries to obtain entity type density features and / or keyword granularity features, including:

[0148] S701. Perform named entity recognition on the natural language query statement and extract at least one of the following: organization name entity, year entity, and business terms entity.

[0149] In this application embodiment, named entity recognition refers to natural language processing technology that identifies and categorizes entities with specific meanings from unstructured text. Organization name entities refer to text fragments representing the names of relevant organizations or departments; year entities refer to text fragments representing specific points in time, time periods, or fiscal years; business term entities refer to professional terms with specific referential meanings in the field of archives, such as "audit report," "meeting minutes," and "as-built drawings."

[0150] In some embodiments of this application, the system invokes a deep learning model (e.g., a BERT-Conditional Random Field (CRF) model or a Bidirectional Long Short-Term Memory-Conditional Random Field (BiLSTM-CRF) model) pre-tuned based on an archival domain corpus to perform sequence labeling on natural language query statements. This model can identify character boundaries in natural language query statements and label them as corresponding entity categories. For example, for the input "XX City XX Bureau 2022 Student Status Management Regulations", the system identifies "XX City XX Bureau" as an organization name entity, "2022" as a year entity, and "Student Status Management Regulations" as a business clause entity.

[0151] S702. Determine the ratio between the number of characters occupied by the organization name entity and / or year entity and the total number of characters in the natural language query statement to obtain the entity type density feature.

[0152] In this embodiment, the entity type density feature is a numerical index used to quantify the density of "hard constraints" in a natural language query. This feature reflects the structured nature of the user's query.

[0153] In some embodiments of this application, in step S702, the system first counts the total number of characters contained in the organization name entity and year entity extracted in step S701. Then, the system obtains the total number of characters in the natural language query statement (selectively removing spaces and punctuation). Finally, the system performs a division operation, dividing the total number of characters occupied by the entity by the total number of characters in the statement; the quotient is the entity type density feature. If this ratio approaches 1, it indicates that the user's query statement is almost entirely composed of limiting metadata (e.g., "2023 Personnel Department"), tending towards a macro-level navigation intent; if the ratio is low, it indicates that the query statement contains a large number of natural language conjunctions or descriptive statements (e.g., "detailed processing methods for..."), tending towards a micro-level detail intent.

[0154] S703. Obtain the inverse document frequency value of the business term entity in the offline semantic index library. Based on the inverse document frequency value, determine the semantic specificity of the business term entity and use it as a keyword granular feature.

[0155] In this embodiment, the inverse document frequency (IVF) value refers to a statistical measure of the prevalence of a word in the entire archival corpus; its value is inversely proportional to the frequency of the word's occurrence in the document set. Semantic specificity refers to the ability of a business clause entity to distinguish specific archival content. Keyword granularity features refer to feature values ​​derived from statistics that characterize whether query terms belong to a general or specific category.

[0156] In some embodiments of this application, in step S703, the system uses the business term entity extracted in step S701 (e.g., "completion acceptance form") as the key to query the pre-stored vocabulary statistics table in the offline semantic index library to obtain the inverse document frequency (IDF) value corresponding to the entity. If the query statement contains multiple business term entities, the maximum or average of their IDF values ​​can be taken. If the obtained IDF value is low, it indicates that the entity is a high-frequency general term in the archive (e.g., "notice"), with low semantic specificity, and the keyword granularity feature is coarse-grained; if the IDF value is high, it indicates that the entity is a rare specialized term (e.g., a specific project code or rare business type), with high semantic specificity, and the keyword granularity feature is fine-grained. The system standardizes this value and outputs it as the keyword granularity feature to help determine whether the user is performing generalized browsing or precise searching.

[0157] In the embodiments of this application, in order to ensure the consistency of the data distribution of the input query intent recognition model, after obtaining the original inverse document frequency values ​​of the business term entities, it is also necessary to perform "Min-MaxNormalization" processing, the specific logic of which is as follows:

[0158] Preset boundary values: During the offline index building phase, the system calculates the inverse document frequency (IDF) of all words and records the maximum IDF in the corpus. max (Usually corresponds to rare words that appear only once) and minimum inverse document frequency (IDF) min (These usually correspond to high-frequency words that are present in all documents).

[0159] Feature transformation: During the online retrieval phase, obtain the original inverse document frequency (IDF) values ​​of the business terms entities to be processed. val Then, the final keyword granularity feature (denoted as G) is calculated using the following formula:

[0160] G=(IDF val -IDF min ) / (IDF max -IDF min )

[0161] Outlier handling: If the calculated G value is less than 0, it is forcibly set to 0; if it is greater than 1, it is forcibly set to 1.

[0162] For example: Assume IDF max =10, IDF min =0. If the original inverse document frequency value of the query term "completion acceptance" is 8, then the keyword granularity feature after normalization is 0.8. This feature value is on the same order of magnitude as the entity type density feature, which has the same value range [0,1]. When both are input into the query intent recognition model, it can be ensured that the contribution of each dimension feature to the model decision is based on information content rather than numerical scale, thereby improving the robustness of intent recognition.

[0163] As can be seen, this application embodiment integrates named entity recognition technology with inverse document frequency statistics to perform multi-dimensional feature quantification analysis on user query statements from two dimensions: syntactic structure and statistical distribution. This scheme can accurately quantify the strength of metadata constraints (entity density) and the specificity of words (keyword granularity) contained in the query statement, thereby providing highly reliable quantitative data support for accurately identifying the user's macro-navigation or micro-search intent and improving the robustness of intent judgment.

[0164] In some embodiments of this application, in order to make the fused semantic vector generated in step S303 more accurate, the step of generating the fused semantic vector corresponding to the fused hierarchical text using a pre-trained language model may include:

[0165] S801. Construct a contrastive learning sample set for fine-tuning the pre-trained language model. The contrastive learning sample set contains positive sample pairs and hard negative sample pairs.

[0166] In this embodiment, the contrastive learning sample set refers to a set of paired data used to train the model to distinguish subtle semantic differences between data. A positive sample pair refers to two text instances that should be mapped to a close distance by the pre-trained language model in the semantic space, representing archival texts with high homology or logical subordination. A negative sample pair refers to two text instances that are highly similar in literal form but completely different in archival management dimensions (such as archival affiliation) and should be distanced in the semantic space.

[0167] In some embodiments of this application, the system can traverse the hierarchical structure of the archival database. When constructing positive sample pairs, the system can select the first and second hierarchical texts under the same volume level (e.g., the same case file), combine the first hierarchical text with its parent node text, and combine the second hierarchical text with its parent node text, thereby forming a positive sample pair. When constructing difficult sample pairs, the system can retrieve archives with similar or identical document level texts (e.g., both titled "Annual Financial Statements") under different archival level texts (e.g., different archival groups), and combine similar archive texts from different archival sources into difficult sample pairs. By introducing difficult sample pairs, the pre-trained language model can focus on the specific features of archival level texts and volume level texts, in addition to general business vocabulary, during the training process.

[0168] S802. By comparing the contrastive learning sample set with the information noise, the loss function is estimated, and the pre-trained language model is fine-tuned for domain adaptation to update the parameters of the pre-trained language model.

[0169] In this embodiment, the Information Noise Contrastive Estimation (InfoNCE) loss function is a mathematical function used to maximize the lower bound of mutual information between positive sample pairs. Domain-adaptive fine-tuning refers to retraining a general language model using domain-specific data to adapt it to the semantic distribution of the archival domain.

[0170] In some embodiments of this application, the system can input positive sample pairs and difficult negative sample pairs into a pre-trained language model. During training, the system can calculate the cosine similarity between the two vectors in a positive sample pair and the cosine similarity between the positive sample and the difficult negative sample. Based on the information-noise contrastive estimation loss function, the system can adjust the model parameters so that the distances between positive sample pairs in the vector space are closer to each other, while the distances between difficult negative sample pairs in the vector space are further apart. This process enables the model to learn the homology features of archives, that is, the model no longer generates vectors based solely on general terms such as "financial" and "report," but encodes implicit structured relationships such as "belonging to the same archival collection" or "belonging to the same case file" into the numerical distribution of the vectors.

[0171] S803. Input the fusion-level text of the archive to be retrieved into the fine-tuned pre-trained language model and output the fusion semantic vector carrying homology features.

[0172] In the embodiments of this application, homology features refer to the mathematical attributes implicit in the fused semantic vector that can characterize that the archive belongs to a specific archival group or a specific case file.

[0173] In some embodiments of this application, the system can concatenate the archival data at the fonds level, catalog level, volume level, document level, and page level in a preset order and input it into the pre-trained language model after fine-tuning in step S802. The system can extract the vector of specific marker bits (such as the [CLS] bit) from the output of the last layer of the model, or perform average pooling on the vectors of all output markers to obtain a fused semantic vector. Due to the use of a contrastive learning strategy, the fused semantic vector generated at this time has clustering characteristics in the vector space. That is, the archival vectors belonging to the same fonds level or volume level will naturally cluster together, while maintaining a clear boundary distance from the archival vectors of other fonds level texts, thereby effectively solving the confusion problem of archival data with the same name across fonds (such as reports with the same name from different companies) during retrieval.

[0174] Furthermore, in step S202, when generating the query semantic vector corresponding to the natural language query statement using the preset pre-trained language model, the fine-tuned pre-trained language model is also used, rather than the pre-trained language model before fine-tuning.

[0175] As can be seen, the embodiments of this application construct specific positive sample pairs and hard negative sample pairs during the model training stage, and optimize the vector generation model using the loss function. This enables the generated fused semantic vector to effectively aggregate files with the same hierarchical affiliation and distinguish files that are literally similar but have different sources. Thus, without relying on explicit rules, the semantic retrieval model improves its ability to distinguish and rank files with the same name across collections.

[0176] In some embodiments of this application, constructing a contrastive learning sample set for fine-tuning a pre-trained language model may include:

[0177] S901. Based on the hierarchical affiliation and semantic entity association of the archives to be retrieved, construct a heterogeneous information network diagram of the archives.

[0178] In this application embodiment, the archival heterogeneous information network graph refers to a directed or undirected graph structure containing multiple types of nodes and multiple types of edges, used to represent the complex multidimensional relationships in the archival system. Node types include, but are not limited to, document nodes, file nodes, fonds nodes, and named entity nodes extracted from the text (such as specific person names and project code nodes). Edge types include "belonging edges" representing physical storage relationships (such as a document belonging to a file) and "mention edges" representing content associations (such as a document mentioning a project code).

[0179] In some embodiments of this application, the system can read the metadata configuration file or database association table of the archive to be retrieved, and construct a hierarchical multi-branch tree structure of the archive with the archival level text as the root node and the page level text as the leaf nodes. Then, the system can use named entity recognition technology to extract key entities in the document level text and establish connections between the document node and the corresponding entity node. In addition, if there are reference relationships in the document level text (e.g., "regarding document No. XX"), the system can also establish "reference edges" between the two involved document nodes. Through this step, the system can transform the flat archive data into a graph rich in semantic topology.

[0180] S902. Perform a meta-path-based random walk in the heterogeneous information network graph of the archives, and determine high-confidence positive sample pairs based on the walk probability.

[0181] In this embodiment, a meta-path refers to a composite sequence of relationships connecting two nodes in a heterogeneous graph, such as "archive file-case file-archive file" (representing a file-file relationship) or "archive file-project entity-archive file" (representing a relationship involving the same project). A random walk refers to the process of starting from a node and hopping through the graph according to a predetermined probability strategy to explore the neighborhood structure. High-confidence positive sample pairs refer to archival text pairs that are closely related in both semantic logic and structural relationships, and have eliminated erroneous archiving noise.

[0182] In some embodiments of this application, the system can define meta-paths with strong semantic constraints, such as "Document A - Same Responsible Person - Same Year - Document B". The system can start from the target document node and perform multi-hop random walks along the preset meta-path to calculate the access probability of reaching other document nodes. The system can set a probability threshold, only considering node pairs with access probabilities higher than this threshold as positive sample pairs. This method can filter out "noise files" that are physically in the same volume but logically unrelated, ensuring that positive sample pairs have substantial business relevance.

[0183] S903. Use a sparse retrieval algorithm to obtain a candidate set with high text similarity to the target archive, and calculate the graph structure distance between each sample in the candidate set and the target archive in the heterogeneous information network graph of the archive, so as to screen out the hard negative sample pairs.

[0184] In this embodiment, sparse retrieval algorithm refers to a keyword matching-based retrieval method, such as the BM25 (Best Matching 25) algorithm. Graph distance refers to the shortest path length or reachability metric between two nodes in a graph.

[0185] In some embodiments of this application, the system can first use the BM25 algorithm to retrieve the top K documents that highly overlap with the target file at the lexical level. Subsequently, the system can calculate the shortest path length between these documents and the target file in the heterogeneous information network graph of the files. If a document has extremely high textual similarity to the target file, but the graph structural distance exceeds a preset safe distance threshold (i.e., they are far apart in the business logic network), the system can classify it as a difficult negative sample pair with high textual similarity but no correlation. Such samples allow the pre-trained language model to not only focus on literal overlap during fine-tuning, but also to learn potential structured semantic differences.

[0186] As can be seen, the embodiments of this application construct a heterogeneous graph containing entities and hierarchical relationships, utilize meta-path walking and graph structure distance verification to accurately eliminate noise interference in physical archiving, and effectively mine adversarial hard-to-bear samples that are literally similar but logically unrelated. This allows the fused semantic vector generated by the fine-tuned pre-trained language model to more accurately reflect the real association distance of archives in the business space, thereby improving the accuracy of retrieval and ranking.

[0187] Secondly, embodiments of this application provide a semantic retrieval ranking system based on archive hierarchy weights. The semantic retrieval ranking system based on archive hierarchy weights runs on an electronic device and is used to execute the semantic retrieval ranking method based on archive hierarchy weights as described in any of the above embodiments.

[0188] Thirdly, embodiments of this application provide an electronic device on which any of the semantic retrieval ranking systems based on archive hierarchy weights provided in the embodiments of this application can be run. The electronic device includes a processor and a memory, the memory storing a computer program configured to be executed by the processor to implement the semantic retrieval ranking method based on archive hierarchy weights as described in any of the above embodiments.

[0189] Fourthly, embodiments of this application provide an electronic device on which any of the semantic retrieval and ranking systems based on archive hierarchy weights provided in embodiments of this application can run. For example... Figure 2 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0190] The electronic device includes a Central Processing Unit (CPU) 201, which can perform various appropriate actions and processes based on a program stored in Read-Only Memory (ROM) 202 or a program loaded from storage portion 208 into Random Access Memory (RAM) 203, such as performing the methods described in the above embodiments. The RAM 203 also stores various programs and data required for system operation. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 204. An Input / Output (I / O) interface 205 is also connected to the bus 204.

[0191] The following components are connected to I / O interface 205: input section 206 including audio input devices, push-button switches, etc.; output section 207 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 208 including a hard disk, etc.; and communication section 209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 209 performs communication processing via a network such as the Internet. Drive 210 is also connected to I / O interface 205 as needed. Removable media 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 210 as needed so that computer programs read from them can be installed into storage section 208 as needed.

[0192] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211. When the computer program is executed by central processing unit (CPU) 201, it performs the various functions defined in this application.

[0193] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0194] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0195] Specifically, the electronic device of this embodiment includes a processor and a memory. The memory is coupled to one or more processors and is used to store computer program code. The computer program code includes computer instructions. One or more processors call the computer instructions to cause the electronic device to perform the method provided in the above embodiment.

[0196] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program configured to be executed by a processor to implement the semantic retrieval ranking method based on archive hierarchy weights as described in any of the above embodiments.

[0197] Sixthly, embodiments of this application provide a computer program product, including a computer program or instructions, which are executed by a processor to implement the semantic retrieval ranking method based on archive hierarchy weights as described in any of the preceding claims.

[0198] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A semantic retrieval and ranking method based on archive hierarchy weights, characterized in that, The semantic retrieval ranking method based on archive hierarchy weights includes: The system obtains a natural language query statement input by the user and multiple files to be retrieved, wherein each file to be retrieved includes multiple levels of text, including archive level text, directory level text, volume level text, file level text and page level text. For each of the aforementioned archives to be retrieved, the hierarchical semantic similarity between the natural language query statement and the archive's archival-level text, catalog-level text, volume-level text, document-level text, and page-level text is determined. Obtain multiple hierarchical weights corresponding to the archive to be retrieved, wherein the multiple hierarchical weights include archival level weight, catalog level weight, volume level weight, document level weight, and page level weight; According to the multiple hierarchical weights corresponding to the file to be retrieved, the hierarchical semantic similarity of the file to be retrieved is weighted and summed to obtain the comprehensive semantic similarity of the file to be retrieved; Based on the comprehensive semantic similarity, the multiple files to be retrieved are sorted to determine the semantic retrieval ranking result of the natural language query statement.

2. The semantic retrieval and ranking method based on archive hierarchy weights as described in claim 1, characterized in that, The step of determining the hierarchical semantic similarity between the natural language query statement and the archival data at the record-level, catalog-level, volume-level, document-level, and page-level, respectively, includes: Call a pre-built offline semantic index library, wherein the offline semantic index library pre-stores hierarchical semantic vectors corresponding to multiple hierarchical texts; Using a pre-trained language model, a query semantic vector corresponding to the natural language query statement is generated; Calculate the cosine similarity value between the query semantic vector and the hierarchical semantic vector of each level of text, and use it as the corresponding hierarchical semantic similarity.

3. The semantic retrieval and ranking method based on archive hierarchy weight as described in claim 2, characterized in that, The offline semantic index library is constructed through the following steps: Parse the hierarchical structure of the archive to be retrieved to determine the parent-child node dependencies among the archival level text, directory level text, volume level text, document level text, and page level text in the archive to be retrieved; According to the parent-child node dependency relationship, the text content of the parent node level is accumulated and concatenated to the beginning of the text content of the child node level to generate the fused level text corresponding to each level of text; Using the pre-trained language model, a fusion semantic vector corresponding to the fusion level text is generated, and this vector serves as the level semantic vector of the corresponding level text, forming the offline semantic index library.

4. The semantic retrieval and ranking method based on archive hierarchy weight as described in claim 3, characterized in that, The step of obtaining the multiple level weights corresponding to the file to be retrieved includes: Semantic features are extracted from the natural language query statement to obtain the entity type density feature and / or keyword granularity feature of the natural language query statement; The entity type density feature and / or the keyword granularity feature are input into a preset query intent recognition model to determine the query intent category of the natural language query statement; In the preset intent-weight mapping table, find the target weight allocation strategy that matches the query intent category; Based on the target weight allocation strategy, the hierarchical weights corresponding to the archival data at the collection level, catalog level, volume level, document level, and page level are determined respectively.

5. The semantic retrieval and ranking method based on archive hierarchy weights as described in claim 4, characterized in that, After determining the semantic retrieval ranking result of the natural language query statement based on the ranking result, the method further includes: Obtain user click behavior data on the semantic search ranking results; Based on the file hierarchy information corresponding to the click behavior data, determine the user's preference characteristics for different file hierarchy information; Based on the preference features, determine the weight adjustment value for each level weight; The target weight allocation strategy in the intent-weight mapping table is updated using the weight correction value.

6. The semantic retrieval and ranking method based on archive hierarchy weight as described in claim 4, characterized in that, The step of finding a target weight allocation strategy that matches the query intent category in a preset intent-weight mapping table includes: If the query intent category is a macro navigation intent, the first weight allocation strategy in the intent-weight mapping table is used as the target weight allocation strategy, wherein the first weight allocation strategy represents that the level weight corresponding to the archival level text and the level weight corresponding to the volume level text are both greater than the level weight corresponding to the document level text and the level weight corresponding to the page level text. If the query intent category is a micro-detail intent, the second weight allocation strategy in the intent-weight mapping table is used as the target weight allocation strategy, wherein the second weight allocation strategy indicates that the level weight corresponding to the archival level text and the level weight corresponding to the volume level text are both less than the level weight corresponding to the document level text and the level weight corresponding to the page level text.

7. The semantic retrieval and ranking method based on archive hierarchy weight as described in claim 4, characterized in that, The step of extracting semantic features from the natural language query statement to obtain the entity type density feature and / or keyword granularity feature of the natural language query statement includes: Named entity recognition is performed on the natural language query statement to extract at least one of the following entities: organization name entity, year entity, and business terms entity; The entity type density feature is obtained by determining the ratio between the number of characters occupied by the organization name entity and / or the year entity and the total number of characters in the natural language query statement; And / or, obtain the inverse document frequency value of the business term entity in the offline semantic index library, determine the semantic specificity of the business term entity based on the inverse document frequency value, and use it as the keyword granular feature.

8. A semantic retrieval and ranking system based on archive hierarchy weights, characterized in that, The semantic retrieval ranking system based on archive hierarchy weights runs on an electronic device and is used to execute the semantic retrieval ranking method based on archive hierarchy weights as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the semantic retrieval ranking method based on archive hierarchy weights as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program configured to be executed by a processor to implement the semantic retrieval ranking method based on archive hierarchy weights as described in any one of claims 1 to 7.

Images