Index optimization and compression storage system and method for large-scale literature set

Abstract

The invention discloses an index optimization and compression storage system and method for a large-scale literature set, and the method comprises the following steps: S1, collecting and preprocessing literature data, and generating a standardized text data set; s2, carrying out keyword semantic vector coding, and constructing a keyword semantic vector matrix; s3, constructing an initial Gaussian mixture model to obtain a clustering center, a covariance matrix and a weight; s4, introducing a sea elephant optimization algorithm to optimize clustering parameters, and outputting an optimal clustering result; s5, constructing a semantic clustering structure, and generating an index tree structure; s6, performing bitmap compression and inverted coding, and constructing an index table supporting Boolean logic; and S7, dynamically accessing the newly added literature, and completing incremental updating of the index structure. The method is used for improving the index construction efficiency and the storage compression rate of a large-scale literature set, and efficient and semantic literature retrieval service capable of being incrementally updated is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval and data compression, and in particular to an index optimization and compression storage system and method for large-scale document collections. Background Art

[0002] Against the backdrop of the rapid development of information technology, large-scale text-based knowledge data, such as academic documents, technical patents, government reports, and corporate research results, is growing exponentially. With the widespread deployment of services such as digital libraries, scientific research databases, knowledge management platforms, patent search systems, and online academic search engines, achieving efficient indexing and compressed storage of such vast collections of documents has become a key technical challenge in document retrieval system design.

[0003] Traditional document indexing methods mostly rely on the inverted index mechanism. An inverted index is a structure that maps keywords appearing in a document to the document number in which they appear, enabling rapid keyword location and document retrieval. However, this mechanism is typically based on shallow text processing and keyword statistics, such as basic methods like term frequency (TF) and inverse document frequency (IDF). These methods ignore the semantic relationships between words and are prone to problems such as word duplication, semantic overlap, or information redundancy. Furthermore, as the total volume of documents continues to expand, the storage space required for the inverted index increases dramatically. With documents ranging from tens of millions to hundreds of millions, its storage efficiency and maintenance costs become bottlenecks that limit system expansion.

[0004] In recent years, with the development of natural language processing technology, researchers have introduced technologies such as word vectors, contextual semantic representations, and deep learning encoders to vectorize document content, thereby attempting to build index structures with greater semantic understanding capabilities. However, these vectorization technologies typically generate high-dimensional dense vectors. In application scenarios targeting large-scale document collections, these dense representations are not only extremely space-intensive but also difficult to directly integrate with traditional inverted index structures. In addition, the deep model training process itself is computationally complex and requires numerous parameters, making it difficult to adapt to the performance requirements of online retrieval systems for lightweight, updatable, and highly responsive performance. Therefore, how to strike a balance between semantic representation capabilities and the compressibility of index structures has become the focus and difficulty of current research on semantic retrieval systems.

[0005] Research on index compression has also proposed methods such as bitmap compression, Bloom filters, and Roaring Bitmap to reduce the redundant representation of document numbers. However, these methods are mostly structural optimizations at the underlying coding level and lack the ability to compress and aggregate the keyword set itself from a semantic perspective. When faced with semantic redundancy such as synonyms and semantically neighboring words, compression technology only plays a role at the data storage level and cannot truly reduce index redundancy at the information level. In addition, most compression methods often require the reconstruction of the entire index structure when faced with dynamic document access or corpus updates, and lack the ability for incremental updates and adaptive learning.

[0006] The Gaussian Mixture Model (GMM), a classic clustering method, is capable of soft clustering samples. It can map keyword semantic vectors to multiple latent semantic categories, providing a mathematical foundation for compressed keyword organization. However, GMM itself relies on initial parameter settings and is prone to falling into local optima. The clustering effect is affected by hyperparameters such as the initial mean, covariance matrix, and mixing weights. Therefore, improving GMM clustering quality has become a key factor affecting the accuracy of semantic clustering.

[0007] In order to overcome the above problems, intelligent optimization algorithms have become an effective means to improve the performance of traditional models in recent years. Among them, animal behavior heuristic algorithms are widely used in model parameter optimization and search tasks because of their strong global search capabilities, few parameters, flexible structure, and easy implementation. As a new intelligent optimization algorithm proposed in 2023, the Walrus Optimization Algorithm (WOA) simulates behaviors such as leader encirclement, spiral raids, and collaborative mutations in a group, achieving a balance between global search and local development, and showing better convergence accuracy and search capabilities. Combining the walrus optimization algorithm with the Gaussian mixture model can effectively solve the problem that the GMM model is sensitive to initial parameters and prone to falling into local optimality, thereby improving the accuracy and stability of keyword semantic clustering.

[0008] Although the above technologies have improved the document indexing problem in some dimensions, they still have the following shortcomings: (1) It is difficult to take into account both semantic modeling capabilities and index structure compressibility at the same time, resulting in low system retrieval efficiency; (2) High-dimensional vector redundancy is difficult to aggregate, and there is a lack of a mechanism to construct a compression structure from a semantic level; (3) Existing clustering methods rely on manually set parameters, the clustering effect is unstable, and there is a lack of optimization control; (4) The index structure lacks dynamic update capabilities and is not suitable for the application needs of the continuous growth of massive documents.

[0009] Therefore, how to provide an index optimization and compression storage system and method for large-scale document collections is an urgent problem that those skilled in the art need to solve. Summary of the Invention

[0010] One purpose of the present invention is to propose an index optimization and compression storage system and method for large-scale document collections. The present invention fully utilizes the semantic clustering capability of the Gaussian mixture model and the global search capability of the walrus optimization algorithm, and describes in detail the entire process from document keyword vector construction, clustering modeling, parameter optimization, semantic index construction to compression encoding and dynamic update. It has the advantages of high clustering accuracy, compact index structure, high storage efficiency and fast retrieval response.

[0011] The index optimization and compression storage method for a large-scale document collection according to an embodiment of the present invention includes the following steps:

[0012] S1. Collect the original text data of a large-scale document collection, pre-process the original text data, and generate a standardized document text dataset;

[0013] S2. Perform vector expression encoding operations on the keywords in the document text dataset, extract the contextual information and semantic features of each keyword, generate a keyword vector set, and construct a keyword semantic vector matrix;

[0014] S3. Build an initial Gaussian mixture model based on the keyword semantic vector matrix, perform soft clustering on the embedded vectors, and obtain the initial cluster centers, covariance matrix, and mixture weight parameters.

[0015] S4. Introduce the walrus optimization algorithm to optimize the clustering parameters of the Gaussian mixture model, encode each individual as a set of clustering parameter combinations, and output the global optimal clustering result;

[0016] S5. Construct a semantic clustering structure based on the optimized clustering results, with each cluster center as a first-level index node and the keyword set within the cluster as a second-level inverted unit, generate semantic compression index units and organize them into an index tree structure;

[0017] S6. Using bitmap compression technology to perform inverted encoding on the document number information in each semantic compression index unit to form a compressed inverted index table that supports Boolean logic retrieval;

[0018] S7. When new document data is accessed, keyword embedding vectors are extracted and input into the optimized Gaussian mixture model to determine the semantic clustering to which they belong. The corresponding index nodes are updated to complete the incremental update of the index structure. The updated index structure is then deployed in the storage system to provide semantic-based fast document retrieval services.

[0019] Optionally, the original text data specifically includes the document title, abstract, keywords and text content, which are used to construct the document semantic index and compressed storage structure.

[0020] Optionally, the preprocessing of the original text data specifically includes text denoising, word segmentation, part-of-speech tagging, stop word removal, lemma restoration and named entity recognition.

[0021] Optionally, the S2 specifically includes:

[0022] S21, perform keyword extraction operation on each document in the document text data set, and obtain the document set D = {d1, d2, ..., d n Each document in} i Keyword set K i ={k i1 ,k i2 ,…,k im};

[0023] S22, for each keyword k ij Perform context window analysis and extract its context word set C under the condition of a given window size w ij ={c ij-w ,…,c ij-1 ,c ij+1 ,…,c ij+w};

[0024] S23. Construct keyword k based on context word set ij The semantic vector representation v ij ∈R d , where d is the dimension of the semantic vector and R is a set of real numbers;

[0025] S24, normalize all keyword semantic vectors in the document collection and construct a keyword semantic vector matrix V = [v1, v2, ..., v p ] T ∈R p×d , where p represents the total number of keywords and d represents the dimension of each vector;

[0026] S25. Using the semantic vector matrix V as the input basis for cluster modeling to complete vector encoding of the semantic features of the document text.

[0027] Optionally, the S3 specifically includes:

[0028] S31, setting the initial number of clusters K, and constructing an initial Gaussian mixture model based on the constructed keyword semantic vector matrix V as input data;

[0029] S32. Initialize the parameter set of Gaussian mixture model in:

[0030] π k represents the mixing weight of the kth Gaussian component, μ krepresents the mean vector of the kth Gaussian component, Σ k represents the covariance matrix of the kth Gaussian component;

[0031] S33, for each semantic vector v i , calculate the posterior probability γ under the kth cluster ik :

[0032]

[0033] Among them, γ ik represents the probability that the i-th semantic vector belongs to the k-th Gaussian component, Represented by the mean vector μ k and the covariance matrix Σ k The multivariate Gaussian distribution with parameter v i The probability density function value at , j represents the index of all Gaussian components;

[0034] S34, iteratively optimizing the Gaussian mixture model parameters using an expectation maximization algorithm, the steps comprising:

[0035] Step E: Calculate the posterior probability γ of each sample based on the current parameters ik ;

[0036] M step: According to γ ik For each Gaussian component, the parameter π k ,μ k ,Σ k Perform maximum likelihood estimation update;

[0037] S35. Set the convergence threshold or the upper limit of the number of iterations. When any condition is met, terminate the iteration and output the final cluster center set of the Gaussian mixture model. Covariance matrix collection With the blending weight set Serves as the basis for optimizing input and index clustering structure construction.

[0038] Optionally, the S4 specifically includes:

[0039] S41, based on the obtained Gaussian mixture model initial parameter set Construct the initial population of the walrus optimization algorithm W={w1,w2,…,w N}, where N represents the population size; w i Represents the i-th individual, coded as That is, a set of Gaussian mixture model parameter combinations represented by the i-th individual;

[0040] S42, define the individual fitness function f(w i) is used to evaluate the clustering quality of the parameter combination represented by each individual on the keyword semantic vector matrix V. The fitness function takes the following form:

[0041]

[0042] Among them, v n represents the nth keyword semantic vector, Individual w i The mixing weight, mean vector and covariance matrix of the k-th Gaussian component in, represents the Gaussian probability density function, p is the total number of vectors;

[0043] S43. In each iteration, the search and update process of the walrus optimization algorithm is executed, including:

[0044] All individuals refer to the optimal individual position w best Perform roundup behavior update, and the position update formula is:

[0045] w i (t+1)=w best (t)-A·|C·w best (t)-w i (t)|;

[0046] Among them, A=2·a·r1-a, C=2·r2, a is the current iteration step control factor, r1, r2 are uniform random numbers, w i (t) represents the solution vector of the i-th individual at the t-th iteration. If the jumping behavior probability condition is met, a spiral jumping search is performed:

[0047] w i (t+1)=w i (t)+b·exp(-l)·cos(2πl);

[0048] Where b is the jump intensity coefficient, l is the spiral parameter random variable, w i (t+1) represents the solution vector of the i-th individual at the t+1-th iteration, and exp is the exponential function;

[0049] S44. After completing the maximum number of iterations T or the fitness converges, output the optimal individual w opt , as the final Gaussian mixture model optimization result, is used to construct the semantic clustering structure.

[0050] Optionally, the S5 specifically includes:

[0051] S51, according to the output optimized Gaussian mixture model clustering result, extracting each cluster center as the first-level index node of the semantic clustering structure;

[0052] S52, classifying keyword vectors belonging to the same cluster into corresponding cluster nodes to form a keyword set corresponding to the cluster;

[0053] S53, organizing the keywords contained in each keyword set according to the document number, and establishing a secondary inverted unit corresponding to the primary index node;

[0054] S54: Establish a mapping relationship between the first-level index node and the corresponding second-level inverted unit to form a semantic compression index unit;

[0055] S55, orderly arranging and structurally managing all semantic compression index units according to the relative distance or prior order between semantic clusters;

[0056] S56. Organize the above structured management results into an index tree structure, wherein the cluster center serves as the main branch node of the index tree and the inverted unit serves as the leaf node, thereby forming a keyword index system with a multi-layer semantic clustering structure.

[0057] Optionally, the S6 specifically includes:

[0058] S61, extracting the document number information contained in each semantic compression index unit, and establishing a mapping relationship between the keyword and the document number;

[0059] S62, sorting the document number set corresponding to each keyword to generate a reverse document number list sorted in ascending order by document number;

[0060] S63. Based on the inverted document number list, construct a corresponding bitmap vector, where each bit in the bitmap vector indicates whether a document number contains a keyword;

[0061] S64, compressing the bitmap vector and generating a bitmap index data structure in a compressed format using a bitmap compression technology;

[0062] S65: Bind the compressed bitmap structure corresponding to each keyword to the semantic compression index unit where it is located to form a keyword-level compressed inverted unit;

[0063] S66. Aggregate all keyword-level compressed inverted index units to establish a compressed inverted index table that supports Boolean logic retrieval operations, thereby achieving efficient positioning and query of keywords in the semantic clustering index structure.

[0064] The index optimization and compression storage system for a large-scale document collection according to an embodiment of the present invention includes the following modules:

[0065] The document processing module is used to collect original documents and complete word segmentation, denoising and restoration to generate a normalized text dataset;

[0066] Keyword encoding module, used to extract keyword context semantic information and generate keyword semantic vector matrix;

[0067] Gaussian modeling module, used to build the initial Gaussian mixture model and output cluster centers, covariance matrix and mixture weights;

[0068] The optimization module is used to adjust the Gaussian model parameters using the walrus optimization algorithm to obtain the optimal clustering results;

[0069] Index building module, used to build semantic compression index units based on clustering results and generate index tree structure;

[0070] A compression module is used to perform bitmap compression on the document numbers in the index unit to generate an inverted index table;

[0071] The update module is used to update the semantic clustering index when new documents are added, thus realizing incremental maintenance of the index.

[0072] The beneficial effects of the present invention are:

[0073] The present invention deeply integrates the Gaussian mixture model with the walrus optimization algorithm to construct a semantic clustering index optimization and compression storage method for large-scale document collections, solving prominent problems in the prior art such as high index structure redundancy, weak semantic expression ability, and low update efficiency. In terms of keyword semantic representation, the present invention uses vector expression encoding to accurately extract the contextual semantic features of keywords, effectively improving the semantic perception ability of index construction; in terms of clustering modeling, the Gaussian mixture model is introduced for soft clustering division, avoiding the problems of blurred boundaries and information fragmentation in traditional hard clustering; and in the model optimization link, the present invention uses the walrus optimization algorithm to perform global search and dynamic tuning of clustering parameters, which not only improves the accuracy and stability of clustering results, but also significantly enhances the adaptability of the model under different corpus distributions.

[0074] In addition, in terms of index structure construction, the present invention proposes a semantic compression index structure with cluster centers as first-level nodes and keyword inverted units as second-level nodes, and organizes them through an index tree, effectively reducing the redundancy between keyword indexes and improving the access efficiency of the search path. Combining bitmap compression technology with inverted document number encoding not only further compresses the storage volume, but also improves the execution efficiency during Boolean logic retrieval. In particular, in terms of index maintenance, the present invention designs an incremental update mechanism based on cluster attribution judgment, supports rapid cluster positioning and structural update of newly added documents, and avoids the system load and response delay caused by index reconstruction in traditional methods.

[0075] In summary, the present invention realizes efficient semantic index compression construction, intelligent parameter adaptive optimization and system-level dynamic update. While maintaining high index retrieval performance, it significantly reduces the computational burden and storage pressure of the system in large-scale document data processing, and has beneficial effects such as semantic accuracy, efficient compression, flexible update and strong system scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0077] Figure 1 A flowchart of the index optimization and compression storage method for large-scale document collections proposed by the present invention;

[0078] Figure 2 This is a schematic diagram of the structure of the index optimization and compression storage system for large-scale document collections proposed by the present invention. DETAILED DESCRIPTION

[0079] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0080] refer to Figure 1 , an index optimization and compression storage method for a large-scale document collection, comprising the following steps:

[0081] S1. Collect the original text data of a large-scale document collection, pre-process the original text data, and generate a standardized document text dataset;

[0082] S2. Perform vector expression encoding operations on the keywords in the document text dataset, extract the contextual information and semantic features of each keyword, generate a keyword vector set, and construct a keyword semantic vector matrix;

[0083] S3. Build an initial Gaussian mixture model based on the keyword semantic vector matrix, perform soft clustering on the embedded vectors, and obtain the initial cluster centers, covariance matrix, and mixture weight parameters.

[0084] S4. Introduce the walrus optimization algorithm to optimize the clustering parameters of the Gaussian mixture model, encode each individual as a set of clustering parameter combinations, and output the global optimal clustering result;

[0085] S5. Construct a semantic clustering structure based on the optimized clustering results, with each cluster center as a first-level index node and the keyword set within the cluster as a second-level inverted unit, generate semantic compression index units and organize them into an index tree structure;

[0086] S6. Using bitmap compression technology to perform inverted encoding on the document number information in each semantic compression index unit to form a compressed inverted index table that supports Boolean logic retrieval;

[0087] S7. When new document data is accessed, keyword embedding vectors are extracted and input into the optimized Gaussian mixture model to determine the semantic clustering to which they belong. The corresponding index nodes are updated to complete the incremental update of the index structure. The updated index structure is then deployed in the storage system to provide semantic-based fast document retrieval services.

[0088] In this embodiment, the original text data specifically includes the document title, abstract, keywords and text content, which are used to construct the document semantic index and compressed storage structure.

[0089] In this embodiment, the preprocessing of the original text data specifically includes text denoising, word segmentation, part-of-speech tagging, stop word removal, lemma restoration and named entity recognition.

[0090] In this embodiment, S2 specifically includes:

[0091] S21, perform keyword extraction operation on each document in the document text data set, and obtain the document set D = {d1, d2, ..., d n Each document in} i Keyword set K i ={k i1 ,k i2 ,…,k im};

[0092] S22, for each keyword k ij Perform context window analysis and extract its context word set C under the condition of a given window size w ij ={c ij-w ,…,c ij-1 ,c ij+1 ,…,c ij+w};

[0093] S23. Construct keyword k based on context word set ij The semantic vector representation v ij ∈R d , where d is the dimension of the semantic vector and R is a set of real numbers;

[0094] S24, normalize all keyword semantic vectors in the document collection and construct a keyword semantic vector matrix V = [v1, v2, ..., v p ] T ∈R p×d , where p represents the total number of keywords and d represents the dimension of each vector;

[0095] S25. Using the semantic vector matrix V as the input basis for cluster modeling to complete vector encoding of the semantic features of the document text.

[0096] In this embodiment, S3 specifically includes:

[0097] S31, setting the initial number of clusters K, and constructing an initial Gaussian mixture model based on the constructed keyword semantic vector matrix V as input data;

[0098] S32. Initialize the parameter set of Gaussian mixture model in:

[0099] π k represents the mixing weight of the kth Gaussian component, μ k represents the mean vector of the kth Gaussian component, Σ k represents the covariance matrix of the kth Gaussian component;

[0100] S33, for each semantic vector v i , calculate the posterior probability γ under the kth cluster ik :

[0101]

[0102] Among them, γ ik represents the probability that the i-th semantic vector belongs to the k-th Gaussian component, Represented by the mean vector μ k and the covariance matrix Σ k The multivariate Gaussian distribution with parameter v i The probability density function value at , j represents the index of all Gaussian components;

[0103] S34, iteratively optimizing the Gaussian mixture model parameters using an expectation maximization algorithm, the steps comprising:

[0104] Step E: Calculate the posterior probability γ of each sample based on the current parameters ik ;

[0105] M step: According to γ ik For each Gaussian component, the parameter π k ,μ k ,Σ k Perform maximum likelihood estimation update;

[0106] S35. Set the convergence threshold or the upper limit of the number of iterations. When any condition is met, terminate the iteration and output the final cluster center set of the Gaussian mixture model. Covariance matrix collection With the blending weight set Serves as the basis for optimizing input and index clustering structure construction.

[0107] In this embodiment, the S4 specifically includes:

[0108] S41, based on the obtained Gaussian mixture model initial parameter set Construct the initial population of the walrus optimization algorithm W={w1,w2,…,w N}, where N represents the population size; w i Represents the i-th individual, coded as That is, a set of Gaussian mixture model parameter combinations represented by the i-th individual;

[0109] S42, define the individual fitness function f(w i ) is used to evaluate the clustering quality of the parameter combination represented by each individual on the keyword semantic vector matrix V. The fitness function takes the following form:

[0110]

[0111] Among them, v n represents the nth keyword semantic vector, Individual w i The mixing weight, mean vector and covariance matrix of the k-th Gaussian component in, represents the Gaussian probability density function, p is the total number of vectors;

[0112] S43. In each iteration, the search and update process of the walrus optimization algorithm is executed, including:

[0113] All individuals refer to the optimal individual position w best Perform roundup behavior update, and the position update formula is:

[0114] w i (t+1)=w best (t)-A·|C·w best (t)-w i (t)|;

[0115] Among them, A=2·a·r1-a, C=2·r2, a is the current iteration step control factor, r1, r2 are uniform random numbers, w i (t) represents the solution vector of the i-th individual at the t-th iteration. If the jumping behavior probability condition is met, a spiral jumping search is performed:

[0116] w i (t+1)=w i (t)+b·exp(-l)·cos(2πl);

[0117] Where b is the jump intensity coefficient, l is the spiral parameter random variable, w i (t+1) represents the solution vector of the i-th individual at the t+1-th iteration, and exp is the exponential function;

[0118] S44. After completing the maximum number of iterations T or the fitness converges, output the optimal individual w opt , as the final Gaussian mixture model optimization result, is used to construct the semantic clustering structure.

[0119] In this embodiment, the S5 specifically includes:

[0120] S51, according to the output optimized Gaussian mixture model clustering result, extracting each cluster center as the first-level index node of the semantic clustering structure;

[0121] S52, classifying keyword vectors belonging to the same cluster into corresponding cluster nodes to form a keyword set corresponding to the cluster;

[0122] S53, organizing the keywords contained in each keyword set according to the document number, and establishing a secondary inverted unit corresponding to the primary index node;

[0123] S54: Establish a mapping relationship between the first-level index node and the corresponding second-level inverted unit to form a semantic compression index unit;

[0124] S55, orderly arranging and structurally managing all semantic compression index units according to the relative distance or prior order between semantic clusters;

[0125] S56. Organize the above structured management results into an index tree structure, wherein the cluster center serves as the main branch node of the index tree and the inverted unit serves as the leaf node, thereby forming a keyword index system with a multi-layer semantic clustering structure.

[0126] In this embodiment, S6 specifically includes:

[0127] S61, extracting the document number information contained in each semantic compression index unit, and establishing a mapping relationship between the keyword and the document number;

[0128] S62, sorting the document number set corresponding to each keyword to generate a reverse document number list sorted in ascending order by document number;

[0129] S63. Based on the inverted document number list, construct a corresponding bitmap vector, where each bit in the bitmap vector indicates whether a document number contains a keyword;

[0130] S64, compressing the bitmap vector and generating a bitmap index data structure in a compressed format using a bitmap compression technology;

[0131] S65: Bind the compressed bitmap structure corresponding to each keyword to the semantic compression index unit where it is located to form a keyword-level compressed inverted unit;

[0132] S66. Aggregate all keyword-level compressed inverted index units to establish a compressed inverted index table that supports Boolean logic retrieval operations, thereby achieving efficient positioning and query of keywords in the semantic clustering index structure.

[0133] refer to Figure 2 , an index optimization and compression storage system for large-scale document collections, including the following modules:

[0134] The document processing module is used to collect original documents and complete word segmentation, denoising and restoration to generate a normalized text dataset;

[0135] Keyword encoding module, used to extract keyword context semantic information and generate keyword semantic vector matrix;

[0136] Gaussian modeling module, used to build the initial Gaussian mixture model and output cluster centers, covariance matrix and mixture weights;

[0137] The optimization module is used to adjust the Gaussian model parameters using the walrus optimization algorithm to obtain the optimal clustering results;

[0138] Index building module, used to build semantic compression index units based on clustering results and generate index tree structure;

[0139] A compression module is used to perform bitmap compression on the document numbers in the index unit to generate an inverted index table;

[0140] The update module is used to update the semantic clustering index when new documents are added, thus realizing incremental maintenance of the index.

[0141] Example 1:

[0142] To verify the feasibility of the present invention, the present invention was applied to the construction of a knowledge service platform for a key university library in a certain province. The platform routinely processes over 12 million documents, covering a wide range of fields, including natural sciences, social sciences, engineering technology, and humanities and history. The platform is open to faculty and students on campus and provides functions such as document search, semantic recommendation, keyword navigation, and citation analysis. As the number of newly added documents continues to grow annually, the platform's existing full-text search engine, which is based on a traditional inverted index, has encountered the following problems:

[0143] First, the system index structure is seriously redundant. Since keywords correspond one-to-one with document numbers and there is no semantic aggregation, there is a lack of compression and integration between a large number of repeated keywords or synonyms, and the index data volume expands rapidly, occupying server disk resources. Secondly, the retrieval efficiency has dropped significantly. The traditional keyword matching mechanism does not have the ability to expand semantics, and the user search experience is poor. Under high concurrency or complex Boolean combination queries, the response time generally exceeds 500ms, which seriously affects the service experience. In addition, after each document update, the system needs to regenerate or modify the index structure on a large scale. The index reconstruction time is long and the system maintenance load is heavy. This problem is particularly prominent during the peak usage period of the teaching season.

[0144] To address these pain points, the project team integrated a large-scale document collection index optimization and compression storage method proposed in this paper, based on a fusion of the Walrus optimization algorithm and a Gaussian mixture model, into the library's knowledge platform. The platform's backend runs on a domestic server environment, using Python to implement the algorithm modules and an Elasticsearch cluster for index deployment. The system took three weeks to build.

[0145] First, a structured extraction of existing literature is performed, including titles, abstracts, keywords, and main text. Preprocessing is completed through steps such as word segmentation, part-of-speech analysis, and named entity recognition, obtaining approximately 1.6 million different high-frequency keywords. The system generates keyword semantic vectors through a contextual co-occurrence and position weighting mechanism, with each vector dimension set to 128. Initial clustering is then performed using a Gaussian mixture model. Based on this, an improved walrus optimization algorithm is used to optimize model parameters (including cluster centers, covariance matrix, and weights). Ultimately, the 1.6 million keywords are compressed and clustered into approximately 5,400 semantic clusters.

[0146] Based on the clustering results, a semantic index tree is constructed, with each cluster as a first-level node. Keywords within the cluster and their corresponding document numbers form a second-level inverted structure, and the document numbers are bitmap compressed using Roaring Bitmap. The resulting index structure is deployed in a cluster storage system and provides standard Boolean search and semantic extension interfaces.

[0147] The system test selected 300,000 documents actively searched by users, simulated different scenarios such as keyword search, high-concurrency access, combined query and fuzzy query, and compared the actual performance of the original indexing scheme and the optimized scheme of the present invention in terms of index volume, response time and system resource consumption.

[0148] Table 1 Performance comparison between the traditional method and the method of the present invention in the university library platform

[0149]

[0150] The data in Table 1 clearly demonstrates that the proposed method offers significant advantages over traditional inverted indexing schemes across multiple key performance metrics. First, in terms of total index structure usage, traditional inverted indexing schemes require 11.2GB of storage space. However, the proposed optimization scheme, by introducing a Gaussian mixture model for semantic clustering compression and employing bitmap coding technology, effectively compresses the index structure to 2.7GB, achieving a compression rate of 75.9%. This result demonstrates that the proposed method can significantly reduce index space overhead in large-scale document environments, saving significant storage resources for the system.

[0151] Secondly, in terms of document number compression, traditional methods do not use any compression measures. However, this invention uses Roaring Bitmap to perform bitmap compression on the inverted number list, achieving a compression rate of 81.6%. This high compression rate further enhances the compactness of the index structure and lays the foundation for subsequent improvements in retrieval performance.

[0152] The present invention also excels in retrieval efficiency. The average response time for common keyword searches is reduced from 486ms in traditional solutions to 164ms, a 66.3% improvement. More notably, when processing complex Boolean queries, the response time is significantly reduced from 798ms to 278ms, demonstrating that the present invention, supported by structural optimization and semantic clustering, possesses enhanced combined retrieval processing capabilities.

[0153] The accuracy of the proposed solution for fuzzy semantic expansion queries reached 88.2%, a 26.7% improvement over the 61.5% of the traditional solution. This demonstrates that the introduced semantic clustering structure better preserves the semantic associations between keywords, effectively improving query recall quality and user experience.

[0154] This invention also demonstrates significant performance advantages in dynamic index updates. While traditional solutions take 740 seconds to rebuild or update the index for every 100,000 new documents, this method takes only 123 seconds, increasing update efficiency by more than six times. This is crucial for library platforms to process daily new documents and maintain index freshness.

[0155] Finally, in terms of system concurrent processing capabilities, the traditional system supports a maximum of 750 concurrent users, but the optimized implementation of this invention increases this to 2100, a 180% increase in scalability. This result demonstrates that this invention not only improves the efficiency of single queries, but also significantly enhances the concurrent access capacity of the entire system, making it more adaptable to peak usage scenarios.

[0156] Comprehensive analysis shows that the data in this table fully verifies the technical advantages of the present invention in index compression, retrieval response, semantic accuracy and system scalability, and fully reflects the application value and practical effect of the solution of the present invention in actual large-scale document platforms.

[0157] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. An index optimization and compression storage method for a large-scale document collection, characterized in that: The steps include: S1. Collect the original text data of a large-scale document collection, pre-process the original text data, and generate a standardized document text dataset; S2. Perform vector expression encoding operations on the keywords in the document text dataset, extract the contextual information and semantic features of each keyword, generate a keyword vector set, and construct a keyword semantic vector matrix; S3. Build an initial Gaussian mixture model based on the keyword semantic vector matrix, perform soft clustering on the embedded vectors, and obtain the initial cluster centers, covariance matrix, and mixture weight parameters. S4. Introduce the walrus optimization algorithm to optimize the clustering parameters of the Gaussian mixture model, encode each individual as a set of clustering parameter combinations, and output the global optimal clustering result; S5. Construct a semantic clustering structure based on the optimized clustering results, with each cluster center as the first-level index node and the keyword set within the cluster as the second-level inverted unit, generate semantic compression index units and organize them into an index tree structure; S6. Using bitmap compression technology to perform inverted encoding on the document number information in each semantic compression index unit to form a compressed inverted index table that supports Boolean logic retrieval; S7. When new document data is accessed, keyword embedding vectors are extracted and input into the optimized Gaussian mixture model to determine the semantic clustering to which they belong. The corresponding index nodes are updated to complete the incremental update of the index structure. The updated index structure is then deployed in the storage system to provide semantic-based fast document retrieval services.

2. The index optimization and compression storage method for a large-scale document collection according to claim 1, characterized in that: The original text data specifically includes the document title, abstract, keywords and text content, which are used to construct the document semantic index and compressed storage structure.

3. The index optimization and compression storage method for a large-scale document collection according to claim 1, characterized in that: The preprocessing of the original text data specifically includes text denoising, word segmentation, part-of-speech tagging, stop word removal, morphological restoration and named entity recognition.

4. The index optimization and compression storage method for a large-scale document collection according to claim 1, characterized in that: The S2 specifically includes: S21, perform keyword extraction operation on each document in the document text data set, and obtain the document set D = {d1, d2, ..., d n Each document in i Keyword set K i ={k i1 ,k i2 ,…,k im }; S22, for each keyword k ij Perform context window analysis and extract its context word set C under the condition of given window size w ij ={c ij-w ,…,c ij-1 ,c ij+1 ,…,c ij+w }; S23. Construct keyword k based on context word set ij The semantic vector representation v ij ∈R d , where d is the dimension of the semantic vector and R is a set of real numbers; S24, normalize all keyword semantic vectors in the document collection and construct a keyword semantic vector matrix V = [v1, v2, ..., v p ] T ∈R p×d , where p represents the total number of keywords and d represents the dimension of each vector; S25. Using the semantic vector matrix V as the input basis for cluster modeling to complete vector encoding of the semantic features of the document text.

5. The index optimization and compression storage method for a large-scale document collection according to claim 1, characterized in that: The S3 specifically includes: S31, setting the initial number of clusters K, and constructing an initial Gaussian mixture model based on the constructed keyword semantic vector matrix V as input data; S32. Initialize the parameter set of Gaussian mixture model in: π k represents the mixing weight of the kth Gaussian component, μ k represents the mean vector of the kth Gaussian component, Σ k represents the covariance matrix of the kth Gaussian component; S33, for each semantic vector v i , calculate the posterior probability γ under the kth cluster ik : Among them, γ ik represents the probability that the i-th semantic vector belongs to the k-th Gaussian component, Represented by the mean vector μ k and the covariance matrix Σ k The multivariate Gaussian distribution with parameter v i The probability density function value at , j represents the index of all Gaussian components; S34, iteratively optimizing the Gaussian mixture model parameters using an expectation maximization algorithm, the steps comprising: Step E: Calculate the posterior probability γ of each sample based on the current parameters ik ; M step: According to γ ik For each Gaussian component, the parameter π k ,μ k ,Σ k Perform maximum likelihood estimation update; S35. Set the convergence threshold or the upper limit of the number of iterations. When any condition is met, terminate the iteration and output the final cluster center set of the Gaussian mixture model. Covariance matrix collection With the blending weight set Serves as the basis for optimizing input and index clustering structure construction.

6. The index optimization and compression storage method for a large-scale document collection according to claim 1, characterized in that: The S4 specifically includes: S41, based on the obtained Gaussian mixture model initial parameter set Construct the initial population of the walrus optimization algorithm W={w1,w2,…,w N }, where N represents the population size; w i Represents the i-th individual, coded as That is, a set of Gaussian mixture model parameter combinations represented by the i-th individual; S42, define the individual fitness function f(w i ) is used to evaluate the clustering quality of the parameter combination represented by each individual on the keyword semantic vector matrix V. The fitness function takes the following form: Among them, v n represents the nth keyword semantic vector, Individual w i The mixing weight, mean vector and covariance matrix of the k-th Gaussian component in, represents the Gaussian probability density function, p is the total number of vectors; S43. In each iteration, the search and update process of the walrus optimization algorithm is executed, including: All individuals refer to the optimal individual position w best Perform roundup behavior update, and the position update formula is: w i (t+1)=w best (t)-A·|C·w best (t)-w i (t)|; Among them, A=2·a·r1-a, C=2·r2, a is the current iteration step control factor, r1, r2 are uniform random numbers, w i (t) represents the solution vector of the i-th individual at the t-th iteration. If the jumping behavior probability condition is met, a spiral jumping search is performed: w i (t+1)=w i (t)+b·exp(-l)·cos(2πl); Where b is the jump intensity coefficient, l is the spiral parameter random variable, w i (t+1) represents the solution vector of the i-th individual at the t+1-th iteration, and exp is the exponential function; S44. After completing the maximum number of iterations T or the fitness converges, output the optimal individual w opt , as the final Gaussian mixture model optimization result, is used to construct the semantic clustering structure.

7. The index optimization and compression storage method for a large-scale document collection according to claim 1, characterized in that: The S5 specifically includes: S51, according to the output optimized Gaussian mixture model clustering result, extracting each cluster center as the first-level index node of the semantic clustering structure; S52, classifying keyword vectors belonging to the same cluster into corresponding cluster nodes to form a keyword set corresponding to the cluster; S53, organizing the keywords contained in each keyword set according to the document number, and establishing a secondary inverted unit corresponding to the primary index node; S54: Establish a mapping relationship between the first-level index node and the corresponding second-level inverted unit to form a semantic compression index unit; S55, orderly arranging and structurally managing all semantic compression index units according to the relative distance or prior order between semantic clusters; S56. Organize the above structured management results into an index tree structure, wherein the cluster center serves as the main branch node of the index tree and the inverted unit serves as the leaf node, thereby forming a keyword index system with a multi-layer semantic clustering structure.

8. The index optimization and compression storage method for a large-scale document collection according to claim 1, characterized in that: The S6 specifically includes: S61, extracting the document number information contained in each semantic compression index unit, and establishing a mapping relationship between the keyword and the document number; S62, sorting the document number set corresponding to each keyword to generate a reverse document number list sorted in ascending order by document number; S63. Based on the inverted document number list, construct a corresponding bitmap vector, where each bit in the bitmap vector indicates whether a document number contains a keyword; S64, compressing the bitmap vector and generating a bitmap index data structure in a compressed format using a bitmap compression technology; S65: Bind the compressed bitmap structure corresponding to each keyword to the semantic compression index unit where it is located to form a keyword-level compressed inverted unit; S66. Aggregate all keyword-level compressed inverted index units to establish a compressed inverted index table that supports Boolean logic retrieval operations, thereby achieving efficient positioning and query of keywords in the semantic clustering index structure.

9. An index optimization and compression storage system for a large-scale document collection, an index optimization and compression storage method for a large-scale document collection according to any one of claims 1 to 8, characterized in that: Includes the following modules: The document processing module is used to collect original documents and complete word segmentation, denoising and restoration to generate a normalized text dataset; Keyword encoding module, used to extract keyword context semantic information and generate keyword semantic vector matrix; Gaussian modeling module, used to build the initial Gaussian mixture model and output cluster centers, covariance matrix and mixture weights; The optimization module is used to adjust the Gaussian model parameters using the walrus optimization algorithm to obtain the optimal clustering results; Index building module, used to build semantic compression index units based on clustering results and generate index tree structure; A compression module is used to perform bitmap compression on the document numbers in the index unit to generate an inverted index table; The update module is used to update the semantic clustering index when new documents are added, thus realizing incremental maintenance of the index.

Images