Text content index automatic identification method based on semantics

Abstract

The invention discloses a semantic-based text content index automatic identification method, which relates to the technical field of information retrieval, and comprises the following steps: initializing sparse projection and LSH signature, performing iterative optimization by using a Lagrange duality form and gradient update, adjusting hash digits, obtaining a fragment index through a k-d tree, and constructing an inverted index. According to the method, compression is performed through Delta coding, an index map is constructed based on Jaccard similarity, compression is performed through WebGraph, CSNMF is used in combination with Z-Laplacian regularization, a low-rank basis matrix and a low-rank coding matrix are generated, a compressed inverted index is reconstructed after iterative optimization, and reconstructed inverted index entries are generated. According to the method, through multi-resolution hash table initialization, joint feature optimization, local adaptive quantization and low-rank index reconstruction, the semantic expression ability and the compression effect of an index structure are improved, the index precision and efficiency are improved, and intelligent identification of index content is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval, in particular to a semantic-based automatic text content indexing identification method. Background Art

[0002] With the rapid development of information technology, the scale of text data is growing exponentially. How to efficiently organize, retrieve and analyze massive amounts of text information has become a major challenge in the current field of natural language processing and information retrieval. Traditional text indexing technologies are mainly based on keyword matching or simple statistical features. Although these methods perform well in structured data retrieval, they have significant limitations when dealing with semantic diversity, ambiguity and contextual dependency. The rise of deep learning and semantic embedding technologies has provided new ideas for text indexing. For example, semantic vector representation methods based on pre-trained models such as BERT and SBERT can better capture the deep semantic features of text. At the same time, the introduction of local sensitive hashing, quantization coding and graph compression technologies has further improved the efficiency and scalability of large-scale text indexing.

[0003] However, existing methods still have shortcomings. The compression and reconstruction process of inverted indexes usually relies on static encoding strategies and lacks adaptability to dynamic semantic distribution, which limits the index compression ratio and query efficiency. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a semantic-based automatic identification method for text content indexing to solve the problem that the compression and reconstruction process of inverted index usually relies on static encoding strategies, lacks adaptability to dynamic semantic distribution, and limits the compression ratio and query efficiency of the index.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a semantic-based text content index automatic identification method, which comprises: Collect text data for preprocessing, generate semantically enhanced data, generate semantic vectors using the SBERT model, generate fused feature vectors using the L2 norm and one-hot encoding, initialize a multi-resolution hash table and partition the fused feature vector, generate enhanced features using the MultiRes-RCF network, combine the reduced-dimensional features into a joint feature vector, initialize sparse projection and LSH signatures, construct a joint optimization objective function, perform iterative optimization using the Lagrangian dual form and gradient update, adjust the number of hash bits, and generate an optimized LSH signature set and partition set. Based on the joint feature vector, LVQ is used to perform local adaptive quantization on the semantic vector to generate a quantized semantic vector. The optimized LSH signature set and partition set are combined to form a metadata set. The shard index is obtained through a kd-tree, and an inverted index is constructed. This is compressed using Delta encoding, and an index graph is constructed based on Jaccard similarity. This is compressed using WebGraph. CSNMF is combined with Z-Laplacian regularization to generate a low-rank basis matrix and a low-rank encoding matrix. After iterative optimization, the compressed inverted index is reconstructed to generate reconstructed inverted index entries. Based on the reconstructed inverted index entries, the Seq2Seq model is used to generate natural language tags and push them.

[0007] As a preferred solution of the semantic-based text content indexing automatic identification method of the present invention, wherein: generating a semantic vector by using the SBERT model, initializing the sparse projection and LSH signature, includes: Based on semantically enhanced data, the SBERT model is used to generate semantic vectors. NER is used to extract entities from the semantically enhanced data. The RoBERTa model is used to extract semantic relationships between entities. The cosine similarity of the semantic vectors between entities is set as the weight of the semantic relationship between entities. The Bayesian dimensionality reduction framework is used to reduce the dimension to obtain a reduced dimensionality feature matrix. LDA combined with Dirichlet prior is used for topic modeling to generate topic distribution vectors, which are then spliced ​​into a topic distribution matrix. Based on the semantic vector, the L2 norm is used for normalization. Based on the dimension reduction feature matrix, the Z-score standardization is used for normalization. Based on the entity relationship, it is converted into a sparse matrix through one-hot encoding. A fused feature vector is generated using a linear weighted formula. The K-means algorithm is used to generate O semantic clusters. The 1D Swin Transformer model is used to extract preliminary semantic features from the fused feature vector within the cluster. The semantic importance score of the semantic cluster is calculated by combining the weights of the semantic relationships between entities. Initialize the multi-resolution hash table, set the allocation rules, calculate the hash index for the fused feature vector, and allocate the fused feature vector to generate a multi-resolution partition set. Adjust the scale weight of the MultiRes-RCF network by using the semantic importance score. Combine the fused feature vector and its corresponding semantic vector to adjust the scale weight. Obtain multi-scale features through multiplication and splice them into enhanced feature vectors. Combine them with the reduced dimensionality feature vector matrix to splice into a joint feature vector. The projection matrix is ​​initialized by random normal distribution, multiplied by joint eigenvector, initialized sparse projection, and initialized preliminary LSH signature set.

[0008] As a preferred solution of the semantic-based text content index automatic identification method of the present invention, the step of constructing a joint optimization objective function and generating an optimized LSH signature set and partition set through iterative optimization includes: Based on the initial projection matrix, hash index, joint eigenvector and initial sparse projection, a joint optimization objective function is constructed and converted into Lagrangian dual form; Based on the joint eigenvector and the initialized projection matrix, the projection error gradient is calculated. Based on the initialized sparse projection, the consistency gradient of the preliminary LSH signature set is calculated. The Lagrange multiplier is updated and the maximum number of iterations is reached. The updated sparse projection, LSH signature set, and projection matrix are output. Based on the updated sparse projection, the hash bits of the LSH function are adjusted by the semantic importance score and the weight of the semantic relationship between entities to generate the optimized LSH signature set and optimized partition set.

[0009] As a preferred solution of the semantic-based text content index automatic identification method of the present invention, wherein: based on the joint feature vector, the semantic vector is locally adaptively quantized using LVQ, the shard index is obtained through the kd tree, and the compression is performed through WebGraph, including: Based on the joint feature vector, LVQ is used for local adaptive quantization to generate a quantized semantic vector, and the optimized LSH signature set is updated to obtain the final LSH signature set; Based on the semantic cluster, the final LSH signature set, optimized partition set, semantic importance frequency and timestamp are spliced ​​into the metadata set of the semantic cluster; Based on the quantized semantic vector, the kd tree algorithm is used for sharding to obtain the shard index; Associate the shard index with the metadata set to obtain the updated metadata set; Based on the topic distribution matrix, semantic clusters are mapped using the cosine function, topic labels are assigned through filtering, and an inverted index is generated using the Lucene library of Elasticsearch. Based on the semantic importance score, the comprehensive importance score of the semantic relationship between entities is calculated and incorporated into the updated metadata set to obtain the extended metadata set. The inverted index is compressed by variable-length coding to obtain a compressed inverted index. Build an index graph, compress it through the WebGraph framework, obtain the compressed index graph, and initialize the inverted index sparse matrix.

[0010] As a preferred solution of the semantic-based text content index automatic identification method of the present invention, wherein: the CSNMF is combined with Z-Laplacian regularization to generate a low-rank basis matrix and a low-rank encoding matrix, and the compressed inverted index is reconstructed after iterative optimization to generate reconstructed inverted index entries, including: Using the collaborative sparse decomposition of CSNMF, the time L0 norm constraint of the initialized inverted index sparse matrix is ​​applied to generate a low-rank basis matrix and a low-rank coding matrix. Combined with the Z-Laplacian regularization matrix, an optimization objective function is constructed. By minimizing the objective function, the low-rank basis matrix and the low-rank coding matrix are updated, and the updated low-rank basis matrix and the low-rank coding matrix are output and spliced ​​into a low-rank decomposition matrix set. The inverted index is reconstructed, the comprehensive similarity score is calculated, and the matrix is ​​sorted in descending order to generate the reconstructed inverted index entries.

[0011] As a preferred solution of the semantic-based text content index automatic identification method of the present invention, the method of generating natural language tags based on the reconstructed inverted index entries using a Seq2Seq model includes: Based on the reconstructed inverted index, the Seq2Seq model is used to generate natural language tags and associate them with the reconstructed inverted index entries.

[0012] As a preferred solution of the semantic-based text content index automatic identification method of the present invention, the pushing includes: Based on the reconstructed inverted index entries after semantic tag association, the cosine similarity of the tag semantic vectors is sorted in descending order, the top K are selected, and pushed through the API interface.

[0013] As a preferred solution of the semantic-based text content index automatic identification method of the present invention, the collecting of text data for preprocessing to generate semantically enhanced data includes: Collect text data through the API interface, configure the polyglot module to identify the text language type, and extract the metadata of the text data through NLP to obtain the original text data, and generate semantically enhanced data through preprocessing.

[0014] In a second aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the semantic-based text content indexing automatic identification method as described in the first aspect of the present invention is implemented.

[0015] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the semantic-based text content indexing automatic identification method as described in the first aspect of the present invention.

[0016] The beneficial effects of the present invention are as follows: the present invention initializes sparse projection and LSH signature, uses Lagrange dual form and gradient update for iterative optimization, adjusts the number of hash bits, obtains a shard index through a kd tree, constructs an inverted index, compresses through Delta coding, constructs an index graph based on Jaccard similarity, compresses through WebGraph, uses CSNMF combined with Z-Laplacian regularization to generate a low-rank basis matrix and a low-rank encoding matrix, reconstructs the compressed inverted index after iterative optimization, and generates reconstructed inverted index entries; the semantic expression ability and compression effect of the index structure are improved, the accuracy and efficiency of the index are improved, and the intelligent identification of the index content is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is a flowchart of the semantic-based text content indexing automatic identification method in Example 1.

[0019] Figure 2 Schematic diagram of multi-resolution hashing and feature optimization of the semantic-based text content indexing automatic identification method in Example 1. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0023] Example 1, reference Figure 1 and Figure 2, which is the first embodiment of the present invention, provides a semantic-based text content index automatic identification method, comprising the following steps: S1. Collect text data for preprocessing, generate semantic enhancement data, generate semantic vectors through the SBERT model, generate fused feature vectors through L2 norm and one-hot encoding, initialize the multi-resolution hash table and partition the fused feature vector, generate enhanced features through the MultiRes-RCF network, combine the dimensionality reduction features into a joint feature vector, initialize sparse projection and LSH signature, construct a joint optimization objective function, use Lagrangian dual form and gradient update for iterative optimization, adjust the number of hash bits, and generate the optimized LSH signature set and partition set; Specifically, we collect text data for preprocessing and generate semantically enhanced data, including: Collect text data through the API interface, configure the polyglot module to identify the text language type, and use NLP to extract the metadata of the text data (including publisher and timestamp) to obtain the original text data; Based on the original text data, regular expressions are used to perform noise removal. The noise includes HTML tags, CSS styles, codes, special characters, repeated spaces, and punctuation. Then, the data is converted to UTF-8 encoding using Python, invalid content (such as empty text) is deleted, and the data is segmented to obtain structured text data. Based on structured text data, BERT is used to segment Chinese text in the text language type identification results, and Hugging Face Tokenizer is used to segment text in other language types. HanLP is used to perform part-of-speech tagging (such as nouns) on the segmented words to generate vocabulary sequences. Based on the vocabulary sequence, we used Scikit-learn's CountVectorizer to generate n-gram candidate words. We then used the Chinese Synonym Forest to perform synonym expansion on the n-gram candidate words in Chinese texts and WordNet to perform synonym expansion on the n-gram candidate words in texts of other languages. We used the SBERT model to calculate the cosine similarity between the n-gram candidate words and synonyms. We then set a cosine similarity threshold based on empirical rules and filtered out synonyms with values ​​below the threshold. Based on word frequency and inverse document frequency, weights are assigned to n-gram candidate words and synonyms to generate a semantically enhanced candidate word set. The formula is: , in, To assign weights, is the word frequency, is the document frequency, is the total number of documents; Based on the semantic enhancement candidate word set, n-gram candidate words and synonyms with a cosine similarity greater than the new threshold are screened, merged, and sorted in descending order based on the assigned weights to generate semantic enhancement data.

[0024] The Polyglot module enables automatic multilingual recognition, avoiding the misjudgment problems caused by traditional single-language detection methods. It is particularly suitable for mixed-language data. Regular expressions combined with UTF-8 encoding conversion effectively remove non-text noise such as HTML tags and special characters, improving the purity of subsequent semantic analysis. Through empty text removal and logical segmentation, the original unstructured data is converted into a standardized format suitable for deep learning model input. BERT word segmentation is used for Chinese, and Hugging Face Tokenizer is used for other languages ​​to ensure that the vocabulary boundaries of different languages ​​conform to their grammatical characteristics. HanLP part-of-speech tagging is used to filter key words such as nouns, reduce interference from irrelevant words, and improve the semantic relevance of subsequent n-gram generation. N-gram candidate words retain local word order information, compensating for the defect of traditional bag-of-words models that ignore contextual relationships. The TF-IDF weighting formula highlights high-frequency but non-common key terms and suppresses interference from common words.

[0025] Furthermore, the semantic vector is generated through the SBERT model, and the sparse projection and LSH signature are initialized, including: Based on semantically enhanced data, the SBERT model is used to generate semantic vectors. Entities (such as names) in the semantically enhanced data are extracted using NER. The RoBERTa model is used to extract semantic relationships between entities (such as technology-driving-industry). The cosine similarity of the semantic vectors between entities is set as the weight of the semantic relationship between entities. The Bayesian dimensionality reduction framework is used for dimensionality reduction to obtain a reduced-dimensional feature matrix. LDA combined with Dirichlet priors is then used for topic modeling to generate topic distribution vectors, which are then concatenated into a topic distribution matrix. Based on the semantic vector, L2 norm is used for normalization. Based on the dimension-reduced feature matrix, Z-score normalization is used for normalization. Based on the entity relationship, one-hot encoding is used to convert it into a sparse matrix (including entity relationship sparse vectors). The fused feature vector is generated using a linear weighted formula. The initial cluster number O is set by the elbow rule, and the K-means algorithm is used on the fused feature vector to generate O semantic clusters; Build a 1D Swin Transformer model, including input layer, block layer, downsampling layer and output layer; Train the 1D Swin Transformer model using the UCR Time Series dataset; Based on semantic clusters, the 1D Swin Transformer model is used to extract preliminary semantic features of the fused feature vector within the cluster. The semantic importance score of the semantic cluster is calculated by combining the weights of the semantic relationships between entities. The formula is: , in, Semantic cluster The semantic importance score of is the balance factor, is the entity relationship sparse vector, For the semantic clusters; Initialize a multi-resolution hash table with L resolution levels (based on a hierarchical pyramid strategy and sorted in ascending order), and set allocation rules based on load balancing; The allocation rule includes screening semantic importance scores greater than a semantic importance score threshold (set based on a quantile method) and allocating them to the high-resolution layer, screening semantic importance scores less than the semantic importance score threshold and allocating them to the low-resolution layer, and otherwise allocating them to the intermediate layer; Based on the initialized multi-resolution hash table, the hash index is calculated for the fused feature vector. The formula is: , in, Resolution layer Hash index of is the fusion feature vector, Resolution layer The resolution parameter, is a spatial hash function; Based on the allocation rules, semantic clusters are allocated to generate a multi-resolution partition set; Based on the multi-resolution partition set, the scale weight of the MultiRes-RCF network is adjusted by the semantic importance score, and the formula is: , in, Resolution layer The adjusted scale weight, Semantic cluster Assign to the corresponding resolution layer The semantic importance score of Semantic cluster The semantic importance score of is the number of semantic clusters, Resolution layer The preset weight coefficient of The fused feature vector and its corresponding semantic vector are input into the MultiRes-RCF network, and combined with the adjusted scale weights, multi-scale features are obtained through multiplication operations, which are concatenated into enhanced feature vectors, and combined with the reduced dimension feature vector matrix to form a joint feature vector. Initialize the projection matrix through random normal distribution, combine with the joint eigenvector for multiplication, initialize the sparse projection, and initialize the preliminary LSH signature set. The formula is: , in, is the preliminary LSH signature, an element in the preliminary LSH signature set, is the initial sparse projection result (based on the product of the initial projection matrix and the joint eigenvector), is a locality sensitive hash function.

[0026] The SBERT model is used to generate semantic vectors. Compared with the traditional bag-of-words model or static word vector, it can capture contextual semantic information more accurately. It is combined with RoBERTa to extract entity relationships and calculate weights through cosine similarity, so that the correlation between entities can be quantified, enhancing the fine-grained expression ability of semantic indexing. The combination of Bayesian dimensionality reduction framework and LDA topic modeling effectively reduces data dimensions and retains key semantic features, reducing the complexity of subsequent calculations. L2 norm normalization and Z-score standardization are used to ensure consistency of different feature scales and avoid certain features dominating model training due to excessive numerical ranges. One-hot encoding converts entity relationships into sparse matrices and combines them with linear weighted formulas to generate fused feature vectors, enhancing the expression ability of features while reducing redundant information. K-means clustering and 1D Swin Transformer extracts semantic cluster features and calculates semantic importance scores based on entity relationship weights, so that key semantic clusters can be prioritized and the retrieval efficiency of the index can be improved. A hierarchical pyramid strategy is used to allocate different resolution layers, so that high-importance semantic clusters are allocated to high-resolution layers to improve retrieval accuracy, and low-importance data are allocated to low-resolution layers to reduce storage overhead. By adjusting the scale weights, the network can adaptively enhance the features of semantic clusters of different importance, thereby improving the robustness of the semantic index. The dimensionality reduction feature matrix is ​​combined into a joint feature vector to ensure that the final index contains both the original semantic information and the integration of multi-scale features, thereby improving retrieval accuracy. The projection matrix is ​​initialized with a random normal distribution and combined with the joint feature vector to generate a preliminary LSH signature, so that the hash code can better preserve semantic similarity. The number of hash bits is adjusted through Lagrangian dual optimization to further improve the discrimination of the LSH signature and reduce the false matching rate.

[0027] Furthermore, a joint optimization objective function is constructed to generate optimized LSH signature sets and partition sets through iterative optimization, including: Based on the initialization of the projection matrix, hash index, joint eigenvector and initialization of the sparse projection, the joint optimization objective function is constructed as follows: , in, To jointly optimize the objective function, To initialize the sparse projection set, is the preliminary LSH signature set, is the joint eigenvector, To initialize the projection matrix regularization parameters, is the matrix Frobenius norm, To initialize the projection matrix, is the sparse regularization parameter, is the signature consistency regularization parameter, is the feature index, is the characteristic number; Based on the joint optimization objective function, it is converted into the Lagrangian dual form, and the formula is: , in, and is the Lagrange multiplier; Based on the joint eigenvector and the initialized projection matrix, the projection error gradient is calculated as follows: , in, is the projection error gradient; Based on the initialized sparse projection, the consistency gradient of the preliminary LSH signature set is calculated. The formula is: , in, is the consistency gradient of the preliminary LSH signature set; Based on the gradient, the Lagrange multiplier is updated as follows: , , in, and Update the Lagrange multipliers and ; Based on the initialized sparse projection, the gradient and updated Lagrange multiplier are used to update, and the preliminary LSH signature and the initialized projection matrix are updated. When the maximum number of iterations is reached, it stops (based on the convergence monitoring method) and outputs the updated sparse projection, LSH signature set, and projection matrix. The formula is: , in, Features Iteration The sparse projection result of is the proximal operator, Features Iteration The sparse projection result of is the adaptive learning rate, Features Iteration Lagrange multipliers of ; Based on the updated sparse projection, the number of hash bits of the LSH function is adjusted by the semantic importance score and the weight of the semantic relationship between entities. The formula is: , in, Resolution layer The adjusted hash bit, is the proportionality coefficient, is the weight of the semantic relationship between entities, Multi-resolution partitioning Resolution layer the number of eigenvectors; Based on the adjusted hash bit count, the optimized LSH signature set and optimized partition set are generated. The formula is: , in, Features At the resolution level Optimized LSH signature.

[0028] By introducing multiple regularization terms into the objective function, such as sparsity constraints, projection consistency terms, and matrix norm terms, a multi-objective trade-off is achieved, including maintaining feature sparsity, enhancing signature consistency, and controlling projection complexity. The constraints of the original objective function are explicitly converted into variable terms using the Lagrangian dual method. Joint optimization is performed via gradient descent, accurately modeling the interactions between variables and improving optimization accuracy. Gradients are evaluated for the projection error and the consistency of the LSH signature, respectively, and used jointly to update the Lagrangian multiplier. A dual gradient feedback mechanism enhances iterative stability and convergence speed. The updated gradients and multipliers are combined to iteratively update the preliminary signature and projection matrix. Adaptive learning rates and proximal operators are introduced to control convergence. The representation quality of hash features is improved by adaptively controlling the iteration step size and sparsity. The number of bits in the LSH function is dynamically adjusted based on the semantic score and the strength of the relationship between entities, achieving multi-resolution projection. This significantly enhances the LSH function's ability to perceive and distinguish different semantic levels. The final optimized LSH signature set and multi-resolution partitioning structure are output, ensuring high-precision feature retrieval and matching capabilities while maintaining compressibility.

[0029] S2. Based on the joint feature vector, LVQ is used to perform local adaptive quantization on the semantic vector to generate a quantized semantic vector. The optimized LSH signature set and partition set are combined to form a metadata set. The shard index is obtained through the kd tree, and an inverted index is constructed. The inverted index is compressed through Delta encoding, and an index graph is constructed based on Jaccard similarity. The index graph is compressed through WebGraph. CSNMF is combined with Z-Laplacian regularization to generate a low-rank basis matrix and a low-rank encoding matrix. After iterative optimization, the compressed inverted index is reconstructed to generate reconstructed inverted index entries. Specifically, based on the joint feature vector, LVQ is used to perform local adaptive quantization on the semantic vector, the shard index is obtained through the kd-tree, and compression is performed through WebGraph, including: Based on the joint feature vector, LVQ is used for local adaptive quantization to generate a quantized semantic vector, and the optimized LSH signature set is updated to obtain the final LSH signature set. The formula is: , in, To quantize the semantic vector, is the semantic vector in the joint feature vector, To quantify the granularity, is the quantization codebook size; Based on the semantic cluster, the final LSH signature set, optimized partition set, semantic importance frequency and timestamp are spliced ​​into the metadata set of the semantic cluster; Based on the quantized semantic vector, the kd tree algorithm is used for sharding to obtain the shard index. The formula is: , in, is the shard index, is the sharding dimension, Select a function for the median, is the kd tree algorithm; Associate the shard index with the metadata set to obtain the updated metadata set; Based on the topic distribution matrix, semantic clusters are mapped using the cosine function. Semantic clusters with cosine function values ​​greater than a threshold (set based on a fixed threshold method) are selected. Topic labels (obtained based on LDA topic modeling) are assigned, and an inverted index is generated using Elasticsearch's Lucene library. Based on the semantic importance score, the comprehensive importance score of the semantic relationship between entities is calculated. The formula is: , in, For the comprehensive importance score, is the balance factor, is the weight of the semantic relationship between entities, Score semantic importance; The comprehensive importance score is incorporated into the updated source dataset to obtain the extended metadata dataset; Based on the extended metadata set, the inverted index is compressed by variable-length coding (DeltaEncode) to obtain a compressed inverted index; Define the quantized semantic vector as a node, calculate the Jaccard similarity of the final LSH signature set, filter out nodes whose Jaccard similarity is greater than the Jaccard similarity threshold, establish edges, and define the Jaccard similarity as the edge weight. Build an index graph, compress it through the WebGraph framework, obtain the compressed index graph, and initialize the inverted index sparse matrix. The formula is: , , in, is the Jaccard similarity, and Semantic clusters and semantic clusters The final LSH signature of To initialize the inverted index sparse matrix, is a sparse matrix, The cluster ID set (obtained based on extended metadata).

[0030] Through LVQ quantization based on joint feature vectors, the redundancy of high-dimensional semantic vectors is effectively reduced, and the quantization error is adaptively adjusted, which helps to reduce the computational complexity while maintaining semantic differentiation. The dynamic update of LSH signature set makes similarity retrieval more efficient and accurate, which is suitable for large-scale semantic search scenarios. The metadata set of semantic clusters not only contains quantitative semantic information, but also integrates dynamic attributes such as timestamps and semantic frequency, supporting time series semantic analysis and real-time recommendation applications. Through kd tree sharding, the spatial query efficiency of high-dimensional data can be significantly improved, which is suitable for fast retrieval of large-scale heterogeneous semantic data. The introduction of LDA topic modeling and cosine similarity screening helps to accurately match query intent in massive semantic clusters and reduce irrelevant data interference. Based on Elasticsearch The inverted index generated by the Lucene library, combined with DeltaEncode compression technology, not only improves index retrieval speed but also greatly saves storage resources. By using a comprehensive importance scoring formula to quantify the strength of semantic associations between entities, it can effectively improve the relevance of retrieval ranking and the accuracy of intelligent recommendations. Constructing a sparse index graph using Jaccard similarity can significantly optimize the connection structure of similar nodes and reduce redundant edges. The sparse matrix generated based on the compressed index graph provides an efficient data structure foundation for fast access to the inverted index.

[0031] Furthermore, CSNMF combined with Z-Laplacian regularization is used to generate a low-rank basis matrix and a low-rank encoding matrix. After iterative optimization, the compressed inverted index is reconstructed to generate reconstructed inverted index entries, including: Use CSNMF's collaborative sparse decomposition to initialize the inverted index sparse matrix with a time L0 norm constraint to generate a low-rank basis matrix and a low-rank encoding matrix; Based on the low-rank basis matrix and the low-rank coding matrix, combined with the Z-Laplacian regularization matrix, an optimization objective function is constructed. By minimizing the objective function, the low-rank basis matrix and the low-rank coding matrix are updated. When the maximum number of updates is reached, the update is stopped (based on the early stopping method). The updated low-rank basis matrix and the low-rank coding matrix are output and spliced ​​into a low-rank decomposition matrix set. The formula is: , , in, is the Z-Laplacian regularization matrix, To compress the index graph, is the regularization strength, and are low-rank basis matrix and low-rank encoding matrix respectively, is the L0 norm sparse penalty coefficient, is the transpose, is the regularization term; Based on the low-rank decomposition matrix set, reconstruct the inverted index, calculate the comprehensive similarity score, sort in descending order, and generate the reconstructed inverted index entries. The formula is: W , , Among them, W To reconstruct the inverted index, For matrix reconstruction, and Update the low-rank basis matrix and low-rank encoding matrix respectively, is the comprehensive similarity score, To reconstruct the quantized vector (based on the reconstructed inverted index), is the balance factor, is the comprehensive importance score, and D is the time decay weight.

[0032] By introducing collaborative sparse decomposition, the inverted index matrix retains semantic consistency while maintaining sparsity, effectively improving the discriminative ability of the feature vector. The fusion of Z-Laplacian regularization constraints allows the graph structure information of the data to be introduced into the optimization objective function, maintaining the local continuity and global consistency of the inverted index feature space. By minimizing and updating the objective function, the convergence stability and accuracy of the decomposition results are improved. At the same time, overfitting can be avoided through the early stopping method. Through the reconstruction and comprehensive scoring mechanism of the inverted index, the semantic enhancement and ranking optimization of the original index are achieved. The introduction of a balance factor dynamically adjusts the relationship between semantic quantization and importance weight, making it more adaptable and adjustable.

[0033] S3: Based on the reconstructed inverted index entries, a Seq2Seq model is used to generate natural language tags and push them. Specifically, based on the reconstructed inverted index entries, the Seq2Seq model is used to generate natural language tags, including: Build a Seq2Seq model, including the input layer, encoder layer, decoder layer, and output layer; Use the DailyDialog dataset to train the Seq2Seq model; Based on the reconstructed inverted index, the Seq2Seq model is used to generate natural language tags, which are converted into tag semantic vectors through the SBERT model. The cosine similarity between the tag semantic vector and the semantic vector is calculated using the cosine formula. Natural language tags whose tag semantic vector cosine similarity is greater than the tag semantic vector cosine similarity threshold (set based on historical experience analysis) are screened. The screened natural language tags are associated with the reconstructed inverted index entries through FAISS.

[0034] By using the Seq2Seq model for natural language tag generation, language tags with contextual understanding capabilities are automatically generated to improve the semantic interpretability of the content. Natural language tags are encoded into semantic vectors through SBERT, and their cosine similarity with the original semantic vectors is calculated to ensure the consistency of the generated tags with the original content in the semantic space. By setting a similarity threshold, only tags with high semantic similarity are retained to achieve accurate matching and enhance the robustness of the tagging system.

[0035] Further, push is performed, including: Based on the reconstructed inverted index entries after semantic tag association, the cosine similarity of the tag semantic vectors is sorted in descending order, the top K are selected, and pushed through the API interface.

[0036] By sorting the reconstructed inverted index entries with associated tags, filtering the TopK semantically relevant content and pushing it through API, we can achieve personalized recommendations and precise content distribution, improve user stickiness, and support dynamic content updates and real-time feedback mechanisms.

[0037] This embodiment also provides a computer device suitable for the semantic-based text content index automatic identification method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the semantic-based text content index automatic identification method proposed in the above embodiment.

[0038] The computer device may be a terminal, comprising a processor, memory, a communication interface, a display, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and computer programs. The internal memory provides an environment for the operating system and computer programs stored in the non-volatile storage media. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication. Wireless communication may be achieved via Wi-Fi, a carrier network, NFC (near-field communication), or other technologies. The display of the computer device may be a liquid crystal display or an electronic ink display. The input device may be a touchscreen overlay on the display, buttons, a trackball, or a touchpad on the computer device housing, or an external keyboard, touchpad, or mouse.

[0039] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the semantic-based text content indexing automatic identification method proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0040] In summary, the present invention initializes sparse projection and LSH signature, uses Lagrange dual form and gradient update for iterative optimization, adjusts the number of hash bits, obtains shard index through kd tree, constructs inverted index, compresses through Delta coding, constructs index graph based on Jaccard similarity, compresses through WebGraph, uses CSNMF combined with Z-Laplacian regularization to generate low-rank basis matrix and low-rank encoding matrix, reconstructs the compressed inverted index after iterative optimization, and generates reconstructed inverted index entries; improves the semantic expression ability and compression effect of the index structure, improves the accuracy and efficiency of the index, and realizes intelligent identification of index content.

[0041] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A semantic-based automatic text content indexing method, characterized by: include, Collect text data for preprocessing, generate semantically enhanced data, generate semantic vectors using the SBERT model, generate fused feature vectors using the L2 norm and one-hot encoding, initialize a multi-resolution hash table and partition the fused feature vector, generate enhanced features using the MultiRes-RCF network, combine the reduced-dimensional features into a joint feature vector, initialize sparse projection and LSH signatures, construct a joint optimization objective function, perform iterative optimization using the Lagrangian dual form and gradient update, adjust the number of hash bits, and generate an optimized LSH signature set and partition set. Based on the joint feature vector, LVQ is used to perform local adaptive quantization on the semantic vector to generate a quantized semantic vector. The optimized LSH signature set and partition set are combined to form a metadata set. The shard index is obtained through a kd-tree, and an inverted index is constructed. This is compressed using Delta encoding, and an index graph is constructed based on Jaccard similarity. This is compressed using WebGraph. CSNMF is combined with Z-Laplacian regularization to generate a low-rank basis matrix and a low-rank encoding matrix. After iterative optimization, the compressed inverted index is reconstructed to generate reconstructed inverted index entries. Based on the reconstructed inverted index entries, the Seq2Seq model is used to generate natural language tags and push them.

2. The method for automatic identification of text content index based on semantics according to claim 1, characterized in that: The generation of semantic vectors through the SBERT model and initialization of sparse projection and LSH signature include: Based on semantically enhanced data, the SBERT model is used to generate semantic vectors. NER is used to extract entities from the semantically enhanced data. The RoBERTa model is used to extract semantic relationships between entities. The cosine similarity of the semantic vectors between entities is set as the weight of the semantic relationship between entities. The Bayesian dimensionality reduction framework is used to reduce the dimension to obtain a reduced dimensionality feature matrix. LDA combined with Dirichlet prior is used for topic modeling to generate topic distribution vectors, which are then spliced ​​into a topic distribution matrix. Based on the semantic vector, the L2 norm is used for normalization. Based on the dimension reduction feature matrix, the Z-score standardization is used for normalization. Based on the entity relationship, it is converted into a sparse matrix through one-hot encoding. A fused feature vector is generated using a linear weighted formula. The K-means algorithm is used to generate O semantic clusters. The 1D Swin Transformer model is used to extract preliminary semantic features from the fused feature vector within the cluster. The semantic importance score of the semantic cluster is calculated by combining the weights of the semantic relationships between entities. Initialize the multi-resolution hash table, set the allocation rules, calculate the hash index for the fused feature vector, and allocate the fused feature vector to generate a multi-resolution partition set. Adjust the scale weight of the MultiRes-RCF network by using the semantic importance score. Combine the fused feature vector and its corresponding semantic vector to adjust the scale weight. Obtain multi-scale features through multiplication and splice them into enhanced feature vectors. Combine them with the reduced dimensionality feature vector matrix to splice into a joint feature vector. The projection matrix is ​​initialized by random normal distribution, multiplied by joint eigenvector, initialized sparse projection, and initialized preliminary LSH signature set.

3. The method for automatic identification of text content index based on semantics according to claim 2, characterized in that: The joint optimization objective function is constructed to generate an optimized LSH signature set and partition set through iterative optimization, including: Based on the initial projection matrix, hash index, joint eigenvector and initial sparse projection, a joint optimization objective function is constructed and converted into Lagrangian dual form; Based on the joint eigenvector and the initialized projection matrix, the projection error gradient is calculated. Based on the initialized sparse projection, the consistency gradient of the preliminary LSH signature set is calculated. The Lagrange multiplier is updated and the maximum number of iterations is reached. The updated sparse projection, LSH signature set, and projection matrix are output. Based on the updated sparse projection, the hash bits of the LSH function are adjusted by the semantic importance score and the weight of the semantic relationship between entities to generate the optimized LSH signature set and optimized partition set.

4. The method for automatic identification of text content index based on semantics according to claim 3, wherein: The method uses LVQ to perform local adaptive quantization on the semantic vector based on the joint feature vector, obtains the shard index through the kd tree, and compresses it through WebGraph, including: Based on the joint feature vector, LVQ is used for local adaptive quantization to generate a quantized semantic vector, and the optimized LSH signature set is updated to obtain the final LSH signature set; Based on the semantic cluster, the final LSH signature set, optimized partition set, semantic importance frequency and timestamp are spliced ​​into the metadata set of the semantic cluster; Based on the quantized semantic vector, the kd tree algorithm is used for sharding to obtain the shard index; Associate the shard index with the metadata set to obtain the updated metadata set; Based on the topic distribution matrix, semantic clusters are mapped using the cosine function, topic labels are assigned through filtering, and an inverted index is generated using the Lucene library of Elasticsearch. Based on the semantic importance score, the comprehensive importance score of the semantic relationship between entities is calculated and incorporated into the updated metadata set to obtain the extended metadata set. The inverted index is compressed by variable-length coding to obtain a compressed inverted index. Construct an index graph, compress it through the WebGraph framework, obtain the compressed index graph, and initialize the inverted index sparse matrix.

5. The method for automatic identification of text content index based on semantics according to claim 4, characterized in that: The method uses CSNMF combined with Z-Laplacian regularization to generate a low-rank basis matrix and a low-rank encoding matrix, reconstructs the compressed inverted index after iterative optimization, and generates reconstructed inverted index entries, including: Using the collaborative sparse decomposition of CSNMF, the time L0 norm constraint of the initialized inverted index sparse matrix is ​​applied to generate a low-rank basis matrix and a low-rank coding matrix. Combined with the Z-Laplacian regularization matrix, an optimization objective function is constructed. By minimizing the objective function, the low-rank basis matrix and the low-rank coding matrix are updated, and the updated low-rank basis matrix and the low-rank coding matrix are output and spliced ​​into a low-rank decomposition matrix set. The inverted index is reconstructed, the comprehensive similarity score is calculated, and the matrix is ​​sorted in descending order to generate the reconstructed inverted index entries.

6. The method for automatic identification of text content index based on semantics according to claim 5, characterized in that: The method of generating natural language tags based on the reconstructed inverted index entries using a Seq2Seq model includes: Based on the reconstructed inverted index, the Seq2Seq model is used to generate natural language tags and associate them with the reconstructed inverted index entries.

7. The method for automatic identification of text content index based on semantics according to claim 6, characterized in that: The pushing includes: Based on the reconstructed inverted index entries after semantic tag association, the cosine similarity of the tag semantic vectors is sorted in descending order, the top K are selected, and pushed through the API interface.

8. The method for automatic identification of text content index based on semantics according to claim 7, characterized in that: The collected text data is preprocessed to generate semantically enhanced data, including: Collect text data through the API interface, configure the polyglot module to identify the text language type, and extract the metadata of the text data through NLP to obtain the original text data, and generate semantically enhanced data through preprocessing.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the semantic-based text content index automatic identification method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the semantic-based text content index automatic identification method according to any one of claims 1 to 8 are implemented.

Images