Semantic-based automatic identification of text content index

Abstract

The application discloses a semantic-based automatic identification method for text content index, relates to the technical field of information retrieval, and comprises initialization of sparse projection and LSH signature, iterative optimization by using Lagrange dual form and gradient update, adjustment of hash bit number, obtaining of a sharding index through a k-d tree, construction of an inverted index, compression through Delta coding, construction of an index graph based on Jaccard similarity, compression through WebGraph, generation of a low-rank basis matrix and a low-rank coding matrix by using CSNMF combined with Z-Laplacian regularization, reconstruction of the compressed inverted index after iterative optimization, and generation of a reconstructed inverted index entry. The application improves the semantic expression capability and compression effect of the index structure, improves the precision and efficiency of the index, and realizes intelligent identification of the index content through multi-resolution hash table initialization, joint feature optimization, local adaptive quantization and low-rank index reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information retrieval, in particular to a semantic-based automatic identification method for text content index. BACKGROUND

[0002] With the rapid development of information technology, the scale of text data grows exponentially, and how to efficiently organize, retrieve and analyze massive text information has become an important challenge in the field of natural language processing and information retrieval. Traditional text indexing technology is mainly based on keyword matching or simple statistical features. Although these methods perform well in structured data retrieval, they have significant limitations in handling semantic diversity, polysemy and context dependence. The rise of deep learning and semantic embedding technology provides a new approach to text indexing. For example, the semantic vector representation method based on BERT, SBERT and other pre-trained models can better capture the deep semantic features of text. At the same time, the introduction of local sensitive hashing, quantization encoding and graph compression technology further improves the efficiency and scalability of large-scale text indexing.

[0003] However, the existing methods still have some shortcomings. The compression and reconstruction process of inverted index usually relies on static coding strategies, which lack adaptability to dynamic semantic distribution, limiting the compression ratio and query efficiency of the index. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a semantic-based automatic identification method for text content index, which solves the problem that the compression and reconstruction process of inverted index usually relies on static coding strategies, which lack adaptability to dynamic semantic distribution, limiting the compression ratio and query efficiency of the index.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In the first aspect, the present application provides a semantic-based automatic identification method for text content index, which includes,

[0008] Collecting text data for preprocessing, generating semantic enhanced data, generating semantic vectors through SBERT model, generating fusion feature vectors through L2 norm and one-hot encoding, initializing multi-resolution hash table and partitioning fusion feature vectors, generating enhanced features through MultiRes-RCF network, combining dimensionality reduction features to splice into joint feature vectors, initializing sparse projection and LSH signature, constructing joint optimization objective function, using Lagrange dual form and gradient update for iterative optimization, and adjusting hash bit number to generate optimized LSH signature set and partition set;

[0009] Based on the joint feature vector, the semantic vector is locally self-adaptive quantized using LVQ to generate a quantized semantic vector, combined with the optimized LSH signature set and the partition set, spliced into a metadata set, through k-d tree to obtain a shard index, and construct an inverted index, compressed through Delta coding, and based on Jaccard similarity to construct an index graph, compressed through WebGraph, using CSNMF combined with Z-Laplacian regularization to generate a low-rank basis matrix and a low-rank coding matrix, and after iterative optimization, the compressed inverted index is reconstructed to generate a reconstructed inverted index entry;

[0010] Based on the reconstructed inverted index entry, a natural language label is generated using a Seq2Seq model, and is pushed.

[0011] As a preferred scheme of the semantic-based text content index automatic identification method, wherein: the semantic vector generated by the SBERT model, the initialization of the sparse projection and the LSH signature, includes:

[0012] Based on the semantic enhanced data, the SBERT model is used to generate the semantic vector, the entities in the semantic enhanced data are extracted using NER, the semantic relationship between entities is extracted using the RoBERTa model, and the cosine similarity of the semantic vector between entities is set as the weight of the semantic relationship between entities, the dimensionality reduction is performed using the Bayesian dimensionality reduction framework to obtain a dimensionality reduction feature matrix, and the topic distribution vector is generated by combining the Dirichlet prior through LDA to perform topic modeling, and spliced into a topic distribution matrix;

[0013] Based on the semantic vector, the L2 norm is used for normalization processing, based on the dimensionality reduction feature matrix, the Z-score standardization is used for standardization processing, based on the entity relationship, the one-hot encoding is converted into a sparse matrix, the fusion feature vector is generated through a linear weighting formula, and the K-means algorithm is used to generate O semantic clusters The 1D Swin Transformer model is used for preliminary semantic feature extraction of the intra-cluster fusion feature vector, and the semantic importance score of the semantic cluster is calculated combined with the weight of the semantic relationship between entities;

[0014] Initialize the multi-resolution hash table, set the allocation rule, calculate the hash index of the fusion feature vector, and allocate the fusion feature vector to generate a multi-resolution partition set, adjust the scale weight of the MultiRes-RCF network through the semantic importance score, combine the fusion feature vector and its corresponding semantic vector to adjust the scale weight, obtain the multi-scale feature through multiplication operation, splice into the enhanced feature vector, and splice into the joint feature vector combined with the dimensionality reduction feature vector matrix;

[0015] The projection matrix is initialized by a random normal distribution, multiplication operation is performed in combination with the joint feature vector, the sparse projection is initialized, and the initial LSH signature set is initialized.

[0016] As a preferred scheme of the automatic identification method of the semantic-based text content index, the joint optimization objective function is constructed, and the optimized LSH signature set and the partition set are generated by iterative optimization.

[0017] Based on the initialized projection matrix, the hash index, the joint feature vector and the initialized sparse projection, the joint optimization objective function is constructed, and is converted into a Lagrange dual form.

[0018] Based on the joint feature vector and the initialized projection matrix, the projection error gradient is calculated, based on the initialized sparse projection, the consistency gradient of the initial LSH signature set is calculated, the Lagrange multiplier is updated, and the output is stopped when the maximum iteration number is reached, and the updated sparse projection, the LSH signature set and the projection matrix are output.

[0019] Based on the updated sparse projection, the hash bit number of the LSH function is adjusted by the semantic importance score and the weight of the semantic relationship between entities, and the optimized LSH signature set and the optimized partition set are generated.

[0020] As a preferred scheme of the automatic identification method of the semantic-based text content index, the joint feature vector is used to locally adaptively quantize the semantic vector using LVQ, the piece index is obtained by k-d tree, and the compression is performed by WebGraph.

[0021] Based on the joint feature vector, the locally adaptive quantization is performed using LVQ to generate the quantized semantic vector, and the optimized LSH signature set is updated to obtain the final LSH signature set.

[0022] Based on the semantic cluster, the final LSH signature set, the optimized partition set, the semantic importance frequency division and the timestamp are spliced into the metadata set of the semantic cluster.

[0023] Based on the quantized semantic vector, the k-d tree algorithm is used for fragmentation to obtain the piece index.

[0024] The piece index is associated with the metadata set to obtain the updated metadata set.

[0025] Based on the topic distribution matrix, the semantic cluster is mapped by a cosine function, the theme label is assigned by screening, and the inverted index is generated using the Lucene library of Elasticsearch.

[0026] Based on the semantic importance score, the comprehensive importance score of the semantic relationship between entities is calculated, the metadata set is updated, the extended metadata set is obtained, the inverted index is compressed by variable length coding, and the compressed inverted index is obtained;

[0027] The index graph is constructed, compressed by the WebGraph framework, and the compressed index graph is obtained.

[0028] As a preferred scheme of the semantic-based text content index automatic identification method, wherein: the CSNMF is used to generate a low-rank basis matrix and a low-rank coding matrix, and after iterative optimization, the compressed inverted index is reconstructed to generate a reconstructed inverted index entry, including:

[0029] The CSNMF is used for collaborative sparse decomposition, L0 norm constraint is applied to the initialized inverted index sparse matrix, a low-rank basis matrix and a low-rank coding matrix are generated, a Z-Laplacian regularization matrix is combined to construct an optimization objective function, the low-rank basis matrix and the low-rank coding matrix are updated by minimizing the objective function, and the updated low-rank basis matrix and the low-rank coding matrix are outputted, spliced into a low-rank decomposition matrix set, and the inverted index is reconstructed, the comprehensive similarity score is calculated, and the reconstructed inverted index entry is generated in descending order.

[0030] As a preferred scheme of the semantic-based text content index automatic identification method, wherein: based on the reconstructed inverted index entry, a Seq2Seq model is used to generate a natural language label, including:

[0031] Based on the reconstructed inverted index, a Seq2Seq model is used to generate a natural language label, and the reconstructed inverted index entry is associated.

[0032] As a preferred scheme of the semantic-based text content index automatic identification method, wherein: the pushing includes:

[0033] Based on the reconstructed inverted index entry associated with the semantic label, the label semantic vector cosine similarity is sorted in descending order, the top K is selected, and the API interface is pushed.

[0034] As a preferred scheme of the semantic-based text content index automatic identification method, wherein: the collection of text data for preprocessing generates semantic enhancement data, including:

[0035] Text data is collected through an API interface, a polyglot module is configured to identify the language type of the text, and the meta information of the text data is extracted through NLP to obtain original text data, and semantic enhancement data is generated through preprocessing.

[0036] In a second aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the computer program, when executed by the processor, implements any step of the method for automatically identifying semantic-based text content index according to the first aspect of the present application.

[0037] In a third aspect, the present application provides a computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any step of the method for automatically identifying semantic-based text content index according to the first aspect of the present application.

[0038] The present application has the advantages that: the present application initializes sparse projection and LSH signature, uses Lagrange dual form and gradient update for iterative optimization, adjusts the number of hash bits, obtains a sharding index through a k-d tree, constructs an inverted index, compresses through Delta coding, constructs an index graph based on Jaccard similarity, compresses through WebGraph, uses CSNMF combined with Z-Laplacian regularization to generate a low-rank basis matrix and a low-rank coding matrix, reconstructs the compressed inverted index after iterative optimization to generate a reconstructed inverted index entry; the semantic expression ability and compression effect of the index structure are improved, the precision and efficiency of the index are improved, and intelligent identification of the index content is realized. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0040] Figure 1 The flowchart of the method for automatically identifying semantic-based text content index in embodiment 1.

[0041] Figure 2 The schematic diagram of multi-resolution hash and feature optimization of the method for automatically identifying semantic-based text content index in embodiment 1. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification.

[0043] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present application.

[0044] Secondly, "one embodiment" or "an embodiment" as used herein means an implementation that can include one or more features, structures, or characteristics, but that does not mean that all of the features, structures, or characteristics are included in one particular implementation. The various features, structures, or characteristics can be combined in one or more implementations.

[0045] Embodiment 1, Reference Figure 1 and Figure 2 The first embodiment of the present application provides a semantic-based text content index automatic identification method, including the following steps:

[0046] S1, collect text data for preprocessing, generate semantic enhanced data, generate semantic vectors through SBERT model, generate fusion feature vectors through L2 norm and one-hot encoding, initialize multi-resolution hash table and partition fusion feature vectors, generate enhanced features through MultiRes-RCF network, combine dimension reduction features to splice into joint feature vectors, initialize sparse projection and LSH signature, construct joint optimization objective function, use Lagrange dual form and gradient update for iterative optimization, and adjust the number of hash bits, generate optimized LSH signature set and partition set;

[0047] Specifically, collecting text data for preprocessing, generating semantic enhanced data, including:

[0048] Collecting text data through API interface, configuring polyglot module to identify the language type of the text, and extracting the meta information (including publisher and timestamp) of the text data through NLP, to obtain the original text data;

[0049] Based on the original text data, use regular expression for denoising, the noise includes HTML tags, CSS styles, codes, special characters, repeated spaces and punctuation, and convert to UTF-8 encoding through Python, delete invalid content (such as empty text), and segment to obtain structured text data;

[0050] Based on the structured text data, Chinese text in the text language type recognition result is segmented through BERT, other language type text is segmented through Hugging Face Tokenizer, and word sequence is generated by using HanLP to perform part-of-speech tagging (such as noun) on the segmented text.

[0051] Based on the word sequence, n-gram candidate words are generated using Scikit-learn's CountVectorizer, n-gram candidate words of Chinese text are synonym expanded using Chinese synonym forest, n-gram candidate words of other language type text are synonym expanded using WordNet, cosine similarity of n-gram candidate words and synonyms is calculated using SBERT model, cosine similarity threshold is set based on empirical rule, synonyms less than cosine similarity threshold are screened and deleted;

[0052] Based on term frequency and inverse document frequency, weights are assigned to n-gram candidate words and synonyms, and a semantic enhanced candidate word set is generated, the formula is:

[0053] ,

[0054] Among them, is the assigned weight, is the term frequency, is the document frequency, is the total number of documents;

[0055] Based on the semantic enhanced candidate word set, n-gram candidate words and synonyms greater than the new cosine similarity threshold are screened, merged, and sorted in descending order based on the assigned weight, and semantic enhanced data is generated.

[0056] Multi-language automatic recognition is realized through the Polyglot module, avoiding misjudgment problems caused by traditional single language detection methods, especially suitable for mixed language data, regular expressions combined with UTF-8 encoding conversion effectively remove non-text noise such as HTML tags and special characters, improve the purity of subsequent semantic analysis, through empty text deletion and logical segmentation, the original unstructured data is converted into a standardized format suitable for deep learning model input, BERT segmentation is used for Chinese, and Hugging Face Tokenizer is used for other languages, ensuring that the division of vocabulary boundaries of different languages conforms to their grammatical characteristics, key words such as nouns are screened through HanLP part-of-speech tagging, reducing irrelevant word interference and improving the semantic relevance of subsequent n-gram generation, n-gram candidate words retain local word sequence information, making up for the defects of traditional bag-of-words model ignoring context relationships, and the TF-IDF weight formula highlights high-frequency but non-common key terms and suppresses common word interference.

[0057] Further, semantic vectors are generated through the SBERT model, sparse projection and LSH signature are initialized, including:

[0058] Based on the semantic enhanced data, the SBERT model is used to generate semantic vectors, the NER is used to extract entities (such as names) in the semantic enhanced data, the RoBERTa model is used to extract semantic relationships between entities (such as technology-pushing-industry), and the cosine similarity of the semantic vectors between entities is set as the weight of the semantic relationship between entities, the Bayesian dimension reduction framework is used for dimension reduction to obtain a dimension reduction feature matrix, and the LDA combined with Dirichlet prior is used for topic modeling to generate a topic distribution vector, which is spliced into a topic distribution matrix;

[0059] Based on the semantic vector, L2 norm is used for normalization, based on the dimension reduction feature matrix, Z-score standardization is used for standardization, and based on the entity relationship, one-hot encoding is used to convert into a sparse matrix (including entity relationship sparse vector), and a fusion feature vector is generated through a linear weighting formula;

[0060] The elbow rule is used to set the initial cluster number O, and the K-means algorithm is used on the fusion feature vector to generate O semantic clusters;

[0061] A 1D Swin Transformer model is constructed, including an input layer, a Block layer, a down-sampling layer, and an output layer;

[0062] The UCR Time Series dataset is used to train the 1D Swin Transformer model;

[0063] Based on the semantic clusters, the 1D Swin Transformer model is used to extract the preliminary semantic features of the intra-cluster fusion feature vector, and the semantic importance score of the semantic cluster is calculated by combining the weight of the semantic relationship between entities, and the formula is:

[0064] ,

[0065] Wherein, is the semantic importance score of the semantic cluster , is a balance factor, is an entity relationship sparse vector, is the th semantic cluster;

[0066] A multi-resolution hash table is initialized, including L resolution layers (set based on hierarchical pyramid strategy and sorted in ascending order), and a distribution rule is set based on load balancing;

[0067] The allocation rules include selecting those with semantic importance scores greater than a semantic importance score threshold (set based on quantiles) and allocating them to a high-resolution layer; selecting those with semantic importance scores less than a semantic importance score threshold and allocating them to a low-resolution layer; otherwise, allocating them to an intermediate layer.

[0068] Based on the initialization of the multi-resolution hash table, the hash index is calculated for the fused feature vector, using the following formula:

[0069] ,

[0070] in, For resolution layer hash index, To fuse feature vectors, For resolution layer Resolution parameters, For space hash function;

[0071] Based on the allocation rules, semantic clusters are allocated to generate a multi-resolution partition set;

[0072] Based on a multi-resolution partition set, the scale weights of the MultiRes-RCF network are adjusted using semantic importance scores, as shown in the formula:

[0073] ,

[0074] in, For resolution layer Adjusting the scale weights, semantic clusters Assign to the corresponding resolution layer semantic importance score, semantic clusters semantic importance score, For the number of semantic clusters, For resolution layer The preset weighting coefficients;

[0075] The fused feature vector and its corresponding semantic vector are input into the MultiRes-RCF network, and multi-scale features are obtained by adjusting the scale weights and multiplying them together to form an enhanced feature vector. This enhanced feature vector is then combined with the reduced feature vector matrix to form a joint feature vector.

[0076] The projection matrix is ​​initialized using a random normal distribution, and then multiplied using the joint eigenvectors to initialize the sparse projection and the initial LSH signature set, as shown in the formula:

[0077] ,

[0078] wherein, is a preliminary LSH signature, is an element in the preliminary LSH signature set, is an initial sparse projection result (based on the product of the initialization projection matrix and the joint feature vector), is a local sensitive hash function.

[0079] The SBERT model is used to generate semantic vectors. Compared with the traditional bag-of-words model or static word vectors, it can more accurately capture the context semantic information. The RoBERTa is used for entity relationship extraction, and the cosine similarity is used to calculate the weight, so that the correlation between entities can be quantified, the fine-grained expression ability of the semantic index is enhanced, the Bayesian dimension reduction framework is combined with the LDA topic modeling, the data dimension is effectively reduced and the key semantic features are preserved, the complexity of subsequent calculation is reduced, the L2 norm normalization and Z-score standardization are used to ensure that different feature scales are consistent, avoid some features being dominated by the model training due to the large value range, the one-hot encoding is used to convert the entity relationship into a sparse matrix, and the linear weighting formula is used to generate a fusion feature vector, which enhances the expression ability of the feature and reduces the redundant information. The semantic cluster features are extracted by K-means clustering and 1D Swin Transformer, and the semantic importance score is calculated by combining the entity relationship weight, so that the key semantic clusters can be processed preferentially, and the retrieval efficiency of the index is improved. A hierarchical pyramid strategy is used to allocate different resolution layers, so that high importance semantic clusters are allocated to high resolution layers to improve retrieval accuracy, and low importance data is allocated to low resolution layers to reduce storage overhead. By adjusting the scale weight, the network can adaptively enhance the features of semantic clusters with different importance, improve the robustness of the semantic index, and combine the dimension reduction feature matrix to splice the joint feature vector, so that the final index contains both original semantic information and multi-scale features, improves the retrieval accuracy, and initializes the projection matrix with random normal distribution to generate preliminary LSH signatures with joint feature vectors, so that the hash code can better preserve semantic similarity. The number of hash bits is adjusted by Lagrange dual optimization to further improve the discrimination of LSH signatures and reduce the false match rate.

[0080] Further, a joint optimization objective function is constructed to generate an optimized LSH signature set and a partition set through iterative optimization, including:

[0081] Based on the initialization projection matrix, hash index, joint feature vector, and initialization sparse projection, a joint optimization objective function is constructed, and the formula is:

[0082] ,

[0083] wherein, is a joint optimization objective function, is an initial sparse projection set, is the initial LSH signature set, is the joint feature vector, is the initialization projection matrix regularization parameter, is the matrix Frobenius norm, is the initialization projection matrix, is the sparse regularization parameter, is the signature consistency regularization parameter, is the feature index, is the feature number;

[0084] Based on the joint optimization objective function, it is converted into the Lagrange dual form, and the formula is:

[0085] ,

[0086] wherein, and are the Lagrange multipliers;

[0087] Based on the joint feature vector and the initialization projection matrix, the projection error gradient is calculated, and the formula is:

[0088] ,

[0089] wherein, is the projection error gradient;

[0090] Based on the initialization sparse projection, the consistency gradient of the initial LSH signature set is calculated, and the formula is:

[0091] ,

[0092] wherein, is the consistency gradient of the initial LSH signature set;

[0093] Based on the gradient, the Lagrange multipliers are updated, and the formula is:

[0094] ,

[0095] ,

[0096] wherein, and are the updated Lagrange multipliers and respectively;

[0097] Based on the initialization sparse projection, the gradient and the updated Lagrange multiplier are used for updating, and the preliminary LSH signature and the initialization projection matrix are updated, and the maximum iteration number is stopped (based on the convergence monitoring method setting), and the updated sparse projection, LSH signature set and projection matrix are output, and the formula is:

[0098] ,

[0099] Wherein, is the sparse projection result of the iteration of the feature , is the proximal operator, is the sparse projection result of the iteration of the feature , is the adaptive learning rate, is the Lagrange multiplier of the iteration of the feature .

[0100] Based on the updated sparse projection, the hash bit number of the LSH function is adjusted by the semantic importance score and the weight of the semantic relationship between entities, and the formula is:

[0101] ,

[0102] Wherein, is the adjusted hash bit number of the resolution layer , is the proportion coefficient, is the weight of the semantic relationship between entities, is the feature vector number of the resolution layer of the multi-resolution partition .

[0103] Based on the adjusted hash bit number, the optimized LSH signature set and the optimized partition set are generated, and the formula is:

[0104] ,

[0105] Wherein, is the optimized LSH signature of the feature in the resolution layer .

[0106] By introducing multiple regularization terms such as sparse constraints, projection consistency terms and matrix norm terms into the objective function, multi-objective trade-offs are achieved, including maintaining feature sparsity, enhancing signature consistency and controlling projection complexity. The Lagrange dual method is used to explicitly convert the constraints of the original objective function into variable terms, and joint optimization is performed through gradient descent to improve optimization accuracy. The interaction between variables is accurately modeled, and the gradient of the projection error and the consistency of the LSH signature is evaluated, and is used jointly to update the Lagrange multiplier. The double gradient feedback mechanism enhances the stability and convergence speed of the iteration. The updated gradient and multiplier are used to iteratively update the preliminary signature and projection matrix, and adaptive learning rate and proximal operator are introduced to control the convergence behavior. By adaptively controlling the iteration step size and sparsity, the representation quality of the hash feature is improved. Based on semantic scores and entity relationship strength, the bit number of the LSH function is dynamically adjusted to realize multi-resolution projection, significantly enhancing the perceptual ability and discrimination ability of the LSH function for different semantic levels. The final optimized LSH signature set and multi-resolution partition structure are output, ensuring high-precision feature retrieval and matching ability while maintaining compression.

[0107] S2, based on the joint feature vector, using LVQ to locally adaptively quantize the semantic vector, generate quantized semantic vector, combined with the optimized LSH signature set and partition set, splicing into metadata set, through k-d tree to get the index of the slice, and construct the inverted index, through Delta coding for compression, and based on Jaccard similarity to construct index graph, through WebGraph for compression, using CSNMF combined with Z-Laplacian regularization, generating low-rank basis matrix and low-rank coding matrix, after iterative optimization, the compressed inverted index is reconstructed to generate the reconstructed inverted index entry;

[0108] Specifically, based on the joint feature vector, using LVQ to locally adaptively quantize the semantic vector, through k-d tree to get the index of the slice, through WebGraph for compression, including:

[0109] Based on the joint feature vector, using LVQ to locally adaptively quantize, generate quantized semantic vector, and update the optimized LSH signature set to get the final LSH signature set, the formula is:

[0110] ,

[0111] Wherein, is the quantized semantic vector, is the semantic vector in the joint feature vector, is the quantization granularity, is the size of the quantization codebook;

[0112] Based on semantic clusters, the final LSH signature set, the optimized partition set, the semantic importance frequency division, and the timestamp are spliced into a metadata set of the semantic clusters;

[0113] Based on the quantized semantic vector, the k-d tree algorithm is used for fragmentation to obtain a fragmentation index, and the formula is:

[0114]

[0115] wherein, is the fragmentation index, is the fragmentation dimension, is the median selection function, is the k-d tree algorithm;

[0116] The fragmentation index is associated with the metadata set to obtain an updated metadata set;

[0117] Based on the topic distribution matrix, the semantic clusters are mapped through a cosine function, the semantic cluster cosine function value greater than the semantic cluster cosine function value threshold (based on a fixed threshold method) is screened, a topic label (based on LDA topic modeling) is assigned, and an inverted index is generated using the Lucene library of Elasticsearch;

[0118] Based on the semantic importance score, the comprehensive importance score of the semantic relationship between entities is calculated, and the formula is:

[0119]

[0120] wherein, is the comprehensive importance score, is the balance factor, is the weight of the semantic relationship between entities, is the semantic importance score;

[0121] The comprehensive importance score is incorporated into the updated source data set to obtain an expanded metadata set;

[0122] Based on the expanded metadata set, the inverted index is compressed through variable-length encoding (DeltaEncode encoding) to obtain a compressed inverted index;

[0123] The quantized semantic vector is defined as a node, the Jaccard similarity of the final LSH signature set is calculated, the nodes with Jaccard similarity greater than the Jaccard similarity threshold are screened, edges are established, the Jaccard similarity is defined as the edge weight, an index graph is constructed, the index graph is compressed through the WebGraph framework to obtain a compressed index graph, and an inverted index sparse matrix is initialized, and the formula is:

[0124] ​​​

[0125] ,

[0126] wherein, is the Jaccard similarity, and are the final LSH signatures of semantic cluster and semantic cluster respectively, is the initialized inverted index sparse matrix, is the sparse matrix, is the cluster ID set (based on the extended metadata).

[0127] By LVQ quantization based on the joint feature vector, the redundancy of high-dimensional semantic vectors is effectively reduced, and the quantization error is adaptively adjusted, which helps to reduce the computational complexity while maintaining the semantic discrimination. The dynamic update of the LSH signature set makes the similarity search more efficient and accurate, which is suitable for large-scale semantic search scenarios. The metadata set of the semantic cluster not only contains quantized semantic information, but also integrates dynamic attributes such as timestamps and semantic frequency division, supporting time-series semantic analysis and real-time recommendation applications. Through k-d tree fragmentation, the spatial query efficiency of high-dimensional data can be significantly improved, which is suitable for fast retrieval of large-scale heterogeneous semantic data. The introduction of LDA topic modeling and cosine similarity filtering helps to accurately match the query intent in a large number of semantic clusters and reduce irrelevant data interference. The inverted index generated based on the Elasticsearch Lucene library combined with the DeltaEncode compression technology not only improves the index retrieval speed, but also greatly saves storage resources. Through the comprehensive importance scoring formula, the semantic association strength between entities is quantified, which can effectively improve the relevance of retrieval sorting and the accuracy of intelligent recommendation. By constructing a sparse index graph based on Jaccard similarity, the connection structure of similar nodes can be significantly optimized, and redundant edges can be reduced. The sparse matrix generated based on the compressed index graph provides an efficient data structure basis for fast access of the inverted index.

[0128] Further, CSNMF is used in combination with Z-Laplacian regularization to generate a low-rank basis matrix and a low-rank encoding matrix, and after iterative optimization, the compressed inverted index is reconstructed to generate a reconstructed inverted index entry, including:

[0129] Using the collaborative sparse decomposition of CSNMF, L0 norm constraint is applied to the initialized inverted index sparse matrix to generate a low-rank basis matrix and a low-rank encoding matrix;

[0130] Based on the low-rank basis matrix and the low-rank encoding matrix, an optimization objective function is constructed by combining a Z-Laplacian regularization matrix, the low-rank basis matrix and the low-rank encoding matrix are updated by minimizing the objective function, and the updating is stopped when the maximum number of updates is reached (based on the early stopping method), the updated low-rank basis matrix and the low-rank encoding matrix are output, and the low-rank decomposition matrix set is spliced, and the formula is:

[0131]

[0132]

[0133] wherein, is a Z-Laplacian regularization matrix, is a compressed index graph, is a regularization strength, and are a low-rank basis matrix and a low-rank encoding matrix, is an L0 norm sparse penalty coefficient, is a transpose, is a regularization term.

[0134] Based on the low-rank decomposition matrix set, the inverted index is reconstructed, the comprehensive similarity score is calculated, the descending order is sorted, the reconstructed inverted index entry is generated, and the formula is:

[0135] W

[0136]

[0137] wherein, is a reconstructed inverted index, is a matrix reconstruction, and are an updated low-rank basis matrix and an updated low-rank encoding matrix, is a comprehensive similarity score, is a reconstructed quantization vector (obtained based on the reconstructed inverted index), is a balance factor, is a comprehensive importance score, and D is a time decay weight.

[0138] ​​​​By introducing the collaborative sparse decomposition, the inverted index matrix maintains sparsity while preserving semantic consistency, effectively improving the discriminability of the feature vector. The fusion of Z-Laplacian regularization constraint introduces the graph structure information of the data into the optimization objective function, maintaining the local continuity and global consistency of the inverted index feature space. Through the minimization update of the objective function, the convergence stability and precision of the decomposition result are improved. At the same time, overfitting can be avoided by early stopping method. Through the reconstruction and comprehensive scoring mechanism of the inverted index, semantic enhancement and ranking optimization of the original index are realized. The introduction of the balance factor dynamically regulates between semantic quantization and importance weight, making it more adaptive and adjustable.

[0139] S3, based on the reconstructed inverted index entry, using a Seq2Seq model to generate natural language labels, and pushing;

[0140] Specifically, based on the reconstructed inverted index entry, a Seq2Seq model is used to generate natural language labels, including:

[0141] The Seq2Seq model is constructed, including an input layer, an encoder layer, a decoder layer, and an output layer;

[0142] The Seq2Seq model is trained using the DailyDialog dataset;

[0143] Based on the reconstructed inverted index, a Seq2Seq model is used to generate natural language labels, which are converted into label semantic vectors by an SBERT model. The cosine similarity between the label semantic vectors and the semantic vectors is calculated by the cosine formula. Natural language labels with a label semantic vector cosine similarity greater than a label semantic vector cosine similarity threshold (set based on historical experience analysis) are selected. The filtered natural language labels and the reconstructed inverted index entries are associated by FAISS.

[0144] By using a Seq2Seq model to generate natural language labels, language labels with context understanding ability are automatically generated, improving the content semantic explainability. The natural language labels are encoded into semantic vectors by SBERT, and the cosine similarity between them and the original semantic vectors is calculated to ensure the consistency of the generated labels and the original content in the semantic space. By setting a similarity threshold, only labels with high semantic similarity are retained to achieve precise matching and enhance the robustness of the label system.

[0145] Further, the pushing includes:

[0146] Based on the reconstructed inverted index entry associated with the semantic label, the label semantic vector cosine similarity is sorted in descending order, and the top K is selected. The API interface is used for pushing.

[0147] By sorting the reconstructed inverted index entries of the associated tags, screening TopK semantic related content and pushing through API, personalized recommendation and accurate content distribution are realized, user stickiness is improved, and dynamic content update and real-time feedback mechanism are supported.

[0148] The embodiment further provides a computer device suitable for the case of the automatic identification method of the semantic-based text content index, comprising a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the automatic identification method of the semantic-based text content index proposed in the above embodiment.

[0149] The computer device can be a terminal, and the computer device comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, an operator network, NFC (Near Field Communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0150] The embodiment further provides a storage medium having a computer program stored thereon, the program being executed by a processor to realize the automatic identification method of the semantic-based text content index proposed in the above embodiment. The storage medium can be realized by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk.

[0151] To sum up, the present application uses Lagrange dual form and gradient update to carry out iterative optimization by initializing sparse projection and LSH signature, adjusts the number of hash bits, obtains a fragment index through k-d tree, constructs an inverted index, carries out compression through Delta coding, constructs an index graph based on Jaccard similarity, carries out compression through WebGraph, uses CSNMF combined with Z-Laplacian regularization to generate a low-rank basis matrix and a low-rank coding matrix, reconstructs the compressed inverted index after iterative optimization to generate a reconstructed inverted index entry; the semantic expression ability and compression effect of the index structure are improved, the precision and efficiency of the index are improved, and intelligent identification of the index content is realized.

[0152] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A semantic-based automatic text content indexing and identification method, characterized in that: include, Text data is collected and preprocessed to generate semantically enhanced data. Semantic vectors are generated using the SBERT model. A fused feature vector is generated using L2 norm and one-hot encoding. A multi-resolution hash table is initialized and the fused feature vector is partitioned. Enhanced features are generated using the MultiRes-RCF network. Combined with dimensionality-reduced features, they are concatenated into a joint feature vector. Sparse projection and LSH signatures are initialized. A joint optimization objective function is constructed. Iterative optimization is performed using Lagrange dual form and gradient update. The hash bit length is adjusted to generate an optimized LSH signature set and partition set. The initial multi-resolution hash table includes L resolution layers and sets allocation rules based on load balancing; The allocation rules include selecting those with semantic importance scores greater than a semantic importance score threshold and allocating them to a high-resolution layer; selecting those with semantic importance scores less than a semantic importance score threshold and allocating them to a low-resolution layer; otherwise, allocating them to an intermediate layer. The sparse projection refers to initializing the projection matrix using a random normal distribution, performing multiplication operations with the joint eigenvectors to obtain the initialized sparse projection, and initializing the initial LSH signature set, as shown in the formula: ， in, For the initial LSH signature, it is an element in the initial LSH signature set. This is the initial sparse projection result. For resolution layer hash index, To fuse feature vectors, It is a locality-sensitive hash function; Based on the joint feature vector, LVQ is used to locally adaptively quantize the semantic vector to generate quantized semantic vectors. Combined with the optimized LSH signature set and partition set, they are concatenated into a meta-dataset. The shard index is obtained through kd-tree, and an inverted index is constructed. It is compressed through Delta encoding, and an index graph is constructed based on Jaccard similarity. It is compressed through WebGraph, and CSNMF combined with Z-Laplacian regularization is used to generate a low-rank basis matrix and a low-rank encoding matrix. After iterative optimization, the compressed inverted index is reconstructed to generate reconstructed inverted index entries. The compressed inverted index reconstruction refers to constructing an optimization objective function based on the low-rank basis matrix and low-rank encoding matrix, combined with the Z-Laplacian regularization matrix. The low-rank basis matrix and low-rank encoding matrix are updated by minimizing the objective function, stopping when the maximum number of updates is reached. The updated low-rank basis matrix and low-rank encoding matrix are then output and concatenated into a low-rank decomposition matrix set, as shown in the formula: ， ， in, This is the Z-Laplacian regularization matrix. To compress the index map, For regularization strength, and These are the low-rank basis matrix and the low-rank encoding matrix, respectively. The L0 norm sparsity penalty coefficient, For transpose, For regularization terms, Choose a function for the median; Based on the low-rank decomposition matrix set, the inverted index is reconstructed using the following formula: W ， Among them, W To reconstruct the inverted index, For matrix reconstruction, and These are updates to the low-rank basis matrix and the low-rank encoding matrix, respectively. For cluster ID set; Based on the reconstructed inverted index entries, natural language tags are generated using the Seq2Seq model and then pushed out.

2. The semantic-based automatic text content indexing method as described in claim 1, characterized in that: The process of generating semantic vectors using the SBERT model and initializing sparse projections and LSH signatures includes: Based on semantically augmented data, the SBERT model is used to generate semantic vectors. NER is used to extract entities from the semantically augmented data. The RoBERTa model is used to extract semantic relationships between entities. The cosine similarity of the semantic vectors between entities is set as the weight of the semantic relationship between entities. The Bayesian dimensionality reduction framework is used to reduce the dimensionality and obtain the dimensionality-reduced feature matrix. LDA combined with Dirichlet prior is used to perform topic modeling to generate topic distribution vectors, which are then concatenated into a topic distribution matrix. Based on semantic vectors, L2 norm is used for normalization. Based on the reduced-dimensional feature matrix, Z-score is used for standardization. Based on entity relationships, it is transformed into a sparse matrix through one-hot encoding. A fused feature vector is generated through a linear weighting formula. The K-means algorithm is used to generate O semantic clusters. The 1D Swin Transformer model is used to extract the initial semantic features of the fused feature vectors within the clusters. The semantic importance score of the semantic clusters is calculated by combining the weights of the semantic relationships between entities. Initialize a multi-resolution hash table, set allocation rules, calculate hash indexes for fused feature vectors, allocate fused feature vectors, generate a multi-resolution partition set, adjust the scale weights of the MultiRes-RCF network through semantic importance scoring, combine the fused feature vectors and their corresponding semantic vectors to adjust the scale weights, obtain multi-scale features through multiplication, concatenate them into an enhanced feature vector, and combine them with the dimensionality-reduced feature vector matrix to concatenate them into a joint feature vector. The projection matrix is ​​initialized using a random normal distribution, and the sparse projection is initialized by multiplying the joint eigenvectors. The initial LSH signature set is also initialized.

3. The semantic-based automatic text content indexing method as described in claim 2, characterized in that: The construction of the joint optimization objective function, and the generation of the optimized LSH signature set and partition set through iterative optimization, includes: Based on the initialization projection matrix, hash index, joint eigenvector, and initialization sparse projection, a joint optimization objective function is constructed and converted into a Lagrange dual form. Based on the joint eigenvectors and the initial projection matrix, the projection error gradient is calculated. Based on the initial sparse projection, the initial LSH signature set consistency gradient is calculated. The Lagrange multipliers are updated. The process stops when the maximum number of iterations is reached. The updated sparse projection, LSH signature set, and projection matrix are output. Based on the updated sparse projection, the hash bit length of the LSH function is adjusted by the semantic importance score and the weight of the semantic relationship between entities, thereby generating an optimized LSH signature set and an optimized partition set.

4. The semantic-based automatic text content indexing method as described in claim 3, characterized in that: The process, based on joint feature vectors, uses LVQ for local adaptive quantization of semantic vectors, obtains shard indices through kd-trees, and compresses them using WebGraph, including: Based on the joint feature vector, LVQ is used for local adaptive quantization to generate quantized semantic vectors, and the optimized LSH signature set is updated to obtain the final LSH signature set. Based on semantic clusters, the final LSH signature set, optimized partition set, semantic importance frequency division and timestamp are concatenated into a semantic cluster meta-dataset; Based on quantized semantic vectors, the kd-tree algorithm is used for sharding to obtain sharding indexes; Associate the shard index with the metadata to obtain the updated metadata; Based on the topic distribution matrix, semantic clusters are mapped using a cosine function, topic tags are assigned through filtering, and an inverted index is generated using Elasticsearch's Lucene library. Based on semantic importance scores, a comprehensive importance score of semantic relationships between entities is calculated and incorporated into the updated meta-dataset to obtain an extended meta-dataset. The inverted index is then compressed using variable-length encoding to obtain a compressed inverted index. Construct an index graph, compress it using the WebGraph framework to obtain a compressed index graph, and initialize an inverted index sparse matrix.

5. The semantic-based automatic text content indexing method as described in claim 4, characterized in that: The process involves using CSNMF combined with Z-Laplacian regularization to generate a low-rank basis matrix and a low-rank encoding matrix. After iterative optimization, the compressed inverted index is reconstructed to generate reconstructed inverted index entries, including: Using the collaborative sparse decomposition of CSNMF, an L0 norm constraint is applied to the initial inverted index sparse matrix to generate a low-rank basis matrix and a low-rank encoding matrix. Combined with the Z-Laplacian regularization matrix, an optimization objective function is constructed. By minimizing the objective function, the low-rank basis matrix and the low-rank encoding matrix are updated, and the updated low-rank basis matrix and the low-rank encoding matrix are output. These are then concatenated into a low-rank decomposition matrix set, and the inverted index is reconstructed. The comprehensive similarity score is calculated, sorted in descending order, and the reconstructed inverted index entries are generated.

6. The semantic-based automatic text content indexing method as described in claim 5, characterized in that: The process of generating natural language labels using a Seq2Seq model based on reconstructed inverted index entries includes: Based on the reconstructed inverted index, a Seq2Seq model is used to generate natural language labels, which are then associated with the reconstructed inverted index entries.

7. The semantic-based automatic text content indexing method as described in claim 6, characterized in that: The push notification includes: Based on the reconstructed inverted index entries after semantic tag association, the cosine similarity of the tag semantic vectors is sorted in descending order, and the top K entries are selected and pushed through the API interface.

8. The semantic-based automatic text content indexing method as described in claim 7, characterized in that: The collected text data is preprocessed to generate semantically enhanced data, including: Text data is collected through API interfaces, the polyglot module is configured to identify the language type of the text, and meta-information of the text data is extracted through NLP to obtain the raw text data. Semantic augmentation data is then generated through preprocessing.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the semantic-based text content indexing automatic identification method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the semantic-based text content indexing automatic identification method according to any one of claims 1 to 8.

Images