Document processing method based on dynamic multi-level index and feature clustering

Abstract

The present application relates to the technical field of data processing, and more particularly to a document processing method based on dynamic multi-level index and feature clustering, comprising: capturing a document update event stream in real time, extracting a semantic feature vector when the update frequency variance exceeds a dynamic threshold; constructing a document semantic graph and outputting a graph module degree index; based on the update frequency variance and the graph module degree index, deciding a reconstruction strategy through a reinforcement learning agent to generate a full reconstruction or incremental update instruction; executing index reconstruction in response to the instruction: when full reconstruction, mapping index nodes to graph cluster center compression tree levels, when incremental update, dynamically adjusting displacement node parent pointers; receiving a query request to locate a document cluster, outputting a search result through local sensitive hashing matching; and according to a mismatch case, updating feature extraction model parameters in a closed loop. The present application realizes collaborative optimization of index reconstruction overhead and query efficiency, and reduces system delay in a high-frequency update scenario.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a document processing method based on dynamic multi-level indexing and feature clustering. Background Technology

[0002] In document management systems, the need for storing and retrieving massive amounts of documents has driven the application of dynamic multi-level indexing and feature clustering technologies. Dynamic multi-level indexing automatically reconstructs the index hierarchy based on document update frequency or content changes, achieving adaptive query path optimization. Feature clustering, on the other hand, extracts document feature vectors and applies machine learning algorithms to automatically group similar documents into highly cohesive clusters. After integration, the indexing mechanism can prioritize accessing document clusters defined by feature clustering, reducing irrelevant retrieval operations, ultimately improving document location efficiency and system scalability, and supporting high-precision classification and organization in dynamic environments.

[0003] When dynamic multilevel indexing and feature clustering are applied to document processing, the overhead of dynamic multilevel index reconstruction increases exponentially in environments with high-frequency document updates. When document databases, such as transaction records, frequently add new entries, the index tree is forced to be constantly reconstructed, leading to increased consumption of computing resources and query latency. Feature clustering algorithms rely on the similarity measurement of document feature vectors, but noisy documents, such as emails containing errors or irrelevant content, can cause incorrect clustering assignments, reducing the accuracy of subsequent retrieval. At the same time, integrated systems need to coordinate dynamic index adjustments and clustering optimizations. Real-time document streams, such as the insertion of new content into the platform, may trigger resource contention, causing system response blockages and ultimately dragging down the real-time performance and reliability of document processing. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a document processing method based on dynamic multi-level indexing and feature clustering, which solves the technical problem of increased query latency caused by the increased overhead of dynamic multi-level index reconstruction in high-frequency document update environments.

[0005] To solve the above-mentioned technical problems, the specific details of the present invention are as follows:

[0006] The document processing method based on dynamic multi-level indexing and feature clustering provided by this invention includes:

[0007] Step 1: Obtain the document update event stream, calculate the document update frequency variance in real time, and extract the semantic feature vector of the document in the event stream when the update frequency variance exceeds the dynamic threshold set by the system.

[0008] Step 2: Construct a document semantic graph based on the semantic feature vectors and continuously output the graph modularity index. Input the update frequency variance and the graph modularity index into the reinforcement learning agent. The reinforcement learning agent makes a decision on the reconstruction strategy and generates a reconstruction probability score. If the reconstruction probability score exceeds a preset threshold, a full reconstruction instruction is output; otherwise, an incremental update instruction is output.

[0009] Step 3: Respond to the full reconstruction instruction or the incremental update instruction. When responding to the full reconstruction instruction, map the multi-level index nodes to the cluster center coordinates of the document semantic graph to compress the tree hierarchy. When responding to the incremental update instruction, dynamically adjust the parent pointers of the displacement nodes in the document semantic graph to reduce reconstruction overhead.

[0010] Step 4: Receive external query requests and locate the target cluster in the document semantic graph; filter all document members included in the cluster and extract the feature vectors corresponding to each document member; perform local sensitive hash matching based on the feature vectors and the query request feature vectors; filter document members according to hash matching similarity and output the set of filtered document members as the retrieval result document set;

[0011] Step 5: Calculate the matching confidence of each document in the search results document set with the query request, and filter documents with confidence scores below the threshold as mismatch cases; extract the semantic feature vectors of the mismatch cases and the feature vectors of the query request, and calculate the feature difference value; input the feature difference value into the feature extraction model, and generate the model parameter gradient through backpropagation; update the feature extraction model weights based on the gradient.

[0012] Furthermore, in the document processing method based on dynamic multi-level indexing and feature clustering described in this invention, the step of obtaining the document update event stream includes:

[0013] Obtain the transaction log stream of the document storage system in real time via message queue;

[0014] The transaction log stream is parsed to generate a structured event stream, which includes triplet data of document identifier, timestamp, and original text.

[0015] The variance of the number of updates per unit time in the structured event stream is calculated in real time, and the variance of the number of updates per unit time is used as the basic indicator of the document update frequency variance.

[0016] Furthermore, in the document processing method based on dynamic multi-level indexing and feature clustering described in this invention, step 2 includes:

[0017] Receive a structured event stream and extract the raw text fields from the structured event stream;

[0018] The original text field is input into a pre-trained language model to perform context encoding, and a 128-dimensional dense vector is output as a semantic feature vector.

[0019] The model output process dynamically processes out-of-vocabulary words, retrieves prototype document feature vectors from a pre-defined vector library, calculates the cosine similarity between out-of-vocabulary words and prototype documents, selects the prototype feature vector with the highest similarity to replace the current vector, and outputs the updated semantic feature vector as the input to the graph nodes.

[0020] Furthermore, in the document processing method based on dynamic multi-level indexing and feature clustering described in this invention, step 3 includes:

[0021] The received set of semantic feature vectors is used to define each semantic feature vector as a graph node. The cosine similarity between any two graph nodes is calculated. Undirected edges are created for node pairs with similarity greater than 0.85, generating a relational graph with edge weights.

[0022] The relationship graph is input into a graph convolutional network, and node embedding vectors are generated through a third-order neighbor aggregation operation.

[0023] Perform spectral clustering on the set of node embedding vectors, and output cluster labels and a mapping table between document members and clusters.

[0024] Furthermore, in the document processing method based on dynamic multi-level indexing and feature clustering described in this invention, step 2 includes: constructing the state space of the reinforcement learning agent, inputting the document update frequency variance, inputting the graph modularity index, and inputting the latency change rate of historical query log statistics;

[0025] The action space is defined as a binary decision space. A binary instruction code of 0 represents a full reconstruction instruction, and a binary instruction code of 1 represents an incremental update instruction. The state space input is processed through a policy network to generate reconstruction probability scores.

[0026] Output instruction code 0 when the probability score exceeds 0.7, otherwise output instruction code 1.

[0027] The binary instruction code is pushed to the index reconstruction executor in real time to trigger the reconstruction operation.

[0028] Furthermore, in the document processing method based on dynamic multi-level indexing and feature clustering described in this invention, the reconstruction of the multi-level index structure includes:

[0029] Receive binary instruction codes and parse the instruction code type:

[0030] If the instruction code is 0, read the cluster center coordinates in the document semantic graph generated in step 2 of claim 1, and map the non-leaf nodes of the B+ tree to the coordinates;

[0031] If the instruction code is 1, locate the current displacement node, obtain its displacement path in the member mapping table of claim 4, and dynamically modify the parent pointer;

[0032] Generate a reconstructed index tree file and register the file to the query routing table in step 4 of claim 1 to support document retrieval.

[0033] Furthermore, in the document processing method based on dynamic multi-level indexing and feature clustering described in this invention, the step of performing locality-sensitive hash matching includes:

[0034] The target document cluster is located by reconstructing the index tree file, and the pre-trained language model is reused as an encoder to generate a locality-sensitive hash function.

[0035] Within the target document cluster, extract the feature vectors of each document member in the cluster, calculate the cosine similarity between each feature vector and the feature vector of the query request, filter document members with a cosine similarity greater than 0.85, and output the filtered document member set as the retrieval result document set to step 5.

[0036] Furthermore, in the document processing method based on dynamic multi-level indexing and feature clustering described in this invention, step 5 includes:

[0037] Extract the semantic feature vector of the mismatched document and reuse the query request feature vector as the expected vector;

[0038] Extract the semantic feature vector of the mismatched document and reuse the query request feature vector as the expected vector;

[0039] The Euclidean distance between the feature vector of the mismatched document and the expected vector is calculated as the feature difference value. The feature difference value is then input into the gradient descent optimizer to generate the model parameter adjustment amount.

[0040] The adjustment is backpropagated to the feature extraction model to update the weight parameters, triggering the graph neural network to recalculate the node connections based on the updated parameters.

[0041] Furthermore, the document processing method based on dynamic multi-level indexing and feature clustering described in this invention also includes:

[0042] After the graph neural network outputs cluster labels and member mapping tables, the cluster labels are written to a disk persistent storage file to establish a label library;

[0043] When index reconstruction requires locating cluster center coordinates, the target cluster label is read from the persistent storage file;

[0044] Based on the member document features corresponding to the read cluster labels, a spatial coordinate mapping relationship is constructed through R-tree indexing, and the cluster center coordinates are output to the index reconstruction module.

[0045] Furthermore, the document processing method based on dynamic multi-level indexing and feature clustering described in this invention is characterized by further comprising:

[0046] After generating the document clustering graph in step 2, the graphics rendering engine is invoked to construct a topological visualization view of the graph;

[0047] After calculating the feature difference values ​​of mismatched documents, extract the feature difference value set of all mismatched documents;

[0048] The set of feature difference values ​​is mapped to the corresponding document node positions in the visualization view to generate a dynamically updated heatmap of difference distribution.

[0049] Beneficial effects of this invention;

[0050] This invention addresses the index reconstruction overhead caused by high-frequency document updates through a triple collaborative mechanism. First, a dynamic decision-making mechanism is established, employing a reinforcement learning agent to generate reconstruction strategies based on document update frequency fluctuations and semantic graph stability indicators. This agent analyzes the multi-dimensional state space through a policy network, triggering a full reconstruction instruction when the reconstruction probability score exceeds a set threshold; otherwise, it triggers an incremental update instruction, thus avoiding invalid reconstruction operations caused by high-frequency, minor updates.

[0051] The index structure optimization employs a differentiated execution strategy to respond to decision commands. During the full reconstruction, multi-level index nodes are mapped to the cluster center coordinates of the semantic graph, compressing the tree hierarchy depth to reduce query paths. During incremental updates, the displacement node is located through the member mapping table, and the parent node pointer relationship is dynamically adjusted, changing only the affected subgraphs. The reconstructed index tree is registered to the query routing table to form a streamlined retrieval structure, reducing the reconstruction overhead from a global load to a local operation.

[0052] The closed-loop feedback system drives continuous model optimization through retrieval results. Semantic feature differences between mismatched documents and expected documents are extracted to generate gradient signals, which are then backpropagated to update the feature extraction model parameters. Model weight adjustments trigger the graph neural network to recalculate node connections, improving clustering accuracy and suppressing noise interference. This closed loop reduces the probability of invalid reconstructions caused by clustering errors, ultimately achieving synergistic optimization of index reconstruction overhead and query latency. Attached Figure Description

[0053] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on the accompanying drawings without creative effort.

[0054] Figure 1This is a flowchart illustrating the document processing method based on dynamic multi-level indexing and feature clustering provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the technical solution of the present invention clearer, the present invention will be clearly and completely described below with reference to specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The present invention provided by various embodiments will be described in detail below with reference to the accompanying drawings. To better understand the purpose of the present invention, the present invention will be described in further detail below.

[0056] Please see Figure 1 The document processing method based on dynamic multi-level indexing and feature clustering provided by this invention includes:

[0057] Step 1: Obtain the document update event stream, calculate the document update frequency variance in real time, and extract the semantic feature vector of the document in the event stream when the update frequency variance exceeds the dynamic threshold set by the system.

[0058] Step 2: Construct a document semantic graph based on the semantic feature vectors and continuously output the graph modularity index. Input the update frequency variance and the graph modularity index into the reinforcement learning agent. The reinforcement learning agent makes a decision on the reconstruction strategy and generates a reconstruction probability score. If the reconstruction probability score exceeds a preset threshold, a full reconstruction instruction is output; otherwise, an incremental update instruction is output.

[0059] Step 3: Respond to the full reconstruction instruction or the incremental update instruction. When responding to the full reconstruction instruction, map the multi-level index nodes to the cluster center coordinates of the document semantic graph to compress the tree hierarchy. When responding to the incremental update instruction, dynamically adjust the parent pointers of the displacement nodes in the document semantic graph to reduce reconstruction overhead.

[0060] Step 4: Receive external query requests and locate the target cluster in the document semantic graph; filter all document members included in the cluster and extract the feature vectors corresponding to each document member; perform local sensitive hash matching based on the feature vectors and the query request feature vectors; filter document members according to hash matching similarity and output the set of filtered document members as the retrieval result document set;

[0061] Step 5: Calculate the matching confidence of each document in the search results document set with the query request, and filter documents with confidence scores below the threshold as mismatch cases; extract the semantic feature vectors of the mismatch cases and the feature vectors of the query request, and calculate the feature difference value; input the feature difference value into the feature extraction model, and generate the model parameter gradient through backpropagation; update the feature extraction model weights based on the gradient.

[0062] The document update event stream captures transaction logs from the document storage system in real time via a distributed message queue. Format parsing is performed on the transaction logs to generate a structured event stream consisting of triples including document identifier, timestamp, and original text. The fluctuation characteristics of the update frequency within a unit time window of the structured event stream are statistically analyzed in real time, and its variance is calculated as a basic indicator of document update frequency variance. When this variance exceeds a system dynamic threshold, the semantic feature extraction process is initiated: the original text fields are input into a pre-trained language model, and context encoding is performed through a multi-layer attention mechanism, outputting a dense set of semantic feature vectors with fixed dimensions.

[0063] A set of semantic feature vectors is used as input to a graph neural network to construct a document semantic graph. Each feature vector is mapped to a graph node, and the cosine similarity metric between nodes is calculated. Undirected edges are created for node pairs exceeding a set threshold. The weighted relation graph is input into a graph convolutional network, and node embedding vectors incorporating neighborhood information are generated through multi-order neighbor aggregation operations. Spectral clustering analysis is performed on the set of embedding vectors, outputting cluster labels and a document membership mapping table. The graph modularity stability index is continuously calculated. This modularity index, together with the update frequency variance, constitutes the state space input of the reinforcement learning agent.

[0064] The reinforcement learning agent analyzes state-space data through a neural network policy module to generate index reconstruction probability scores. When the probability score exceeds a decision threshold, a binary full reconstruction instruction code is output; otherwise, an incremental update instruction code is output. The instruction code is transmitted to the index reconstruction module via a high-speed message channel to trigger differentiated operations: during full reconstruction, multi-level index nodes are mapped to the spatial coordinates of the graph cluster centers, compressing the tree hierarchy depth; during incremental updates, the displacement node paths in the member mapping table are located, dynamically adjusting their parent node pointer relationships. The reconstructed index tree file is registered in the query routing table, forming an optimized retrieval path structure.

[0065] Upon receiving an external query request, the system locates the target cluster in the document semantic graph by reconstructing the index tree. A locality-sensitive hash function is generated using the semantic feature extraction model to extract the feature vectors of each document member within the cluster. A similarity index is calculated between the query request's feature vector and the member feature vectors, and document members meeting the matching criteria are selected based on the index value. The selection results are output to the confidence analysis module, which calculates the matching confidence score between each document and the query request.

[0066] Documents with confidence scores below a set threshold are automatically marked as mismatches. The semantic feature vectors of mismatches are extracted and compared with the query request feature vectors, and the Euclidean distance difference between them in the feature space is calculated. This difference value is input into the gradient descent optimizer to generate model parameter adjustments, and the feature extraction model weight parameters are updated through backpropagation. The weight update signal triggers the graph neural network to recalculate node connections, achieving joint optimization of the feature extraction and clustering modules, forming a closed-loop feedback system that suppresses noise interference.

[0067] The technical solution of this invention uses feature extraction to support graph construction, graph indicators to drive reconstruction decisions, index optimization to improve retrieval accuracy, and confidence analysis to provide feedback for model optimization. Through deep collaboration between dynamic indexing and feature clustering, it significantly reduces reconstruction overhead and query latency in high-frequency update scenarios.

[0068] Specifically, in the document processing method based on dynamic multi-level indexing and feature clustering described in this invention, obtaining the document update event stream includes:

[0069] Obtain the transaction log stream of the document storage system in real time via message queue;

[0070] The transaction log stream is parsed to generate a structured event stream, which includes triplet data of document identifier, timestamp, and original text.

[0071] The variance of the number of updates per unit time in the structured event stream is calculated in real time, and the variance of the number of updates per unit time is used as the basic indicator of the document update frequency variance.

[0072] The transaction log stream of the document storage system is captured in real time via a message queue, which employs a distributed architecture to achieve high-throughput data transmission. A format parsing operation is performed on the transaction log stream to extract the document identifier field, timestamp field, and raw text field, generating a structured event stream including triplet data. The fluctuation in the number of updates of the structured event stream within a unit time window is statistically analyzed in real time, and its variance is calculated as a basic indicator of the document update frequency variance. This indicator is output to step 1, providing a quantitative basis for dynamic threshold determination.

[0073] Specifically, in the document processing method based on dynamic multi-level indexing and feature clustering described in this invention, step 2 includes:

[0074] Receive a structured event stream and extract the raw text fields from the structured event stream;

[0075] The original text field is input into a pre-trained language model to perform context encoding, and a 128-dimensional dense vector is output as a semantic feature vector.

[0076] The model output process dynamically processes out-of-vocabulary words, retrieves prototype document feature vectors from a pre-defined vector library, calculates the cosine similarity between out-of-vocabulary words and prototype documents, selects the prototype feature vector with the highest similarity to replace the current vector, and outputs the updated semantic feature vector as the input to the graph nodes.

[0077] After receiving a structured event stream, the original text fields are extracted and input into a pre-trained language model. This language model performs context encoding through a multi-layer attention mechanism, outputting dense vectors of fixed dimensions to represent semantic features. When an out-of-vocabulary word is detected during vector generation, a dynamic compensation mechanism is activated: it queries the prototype document features stored in a pre-defined vector library, calculates the similarity metric between the out-of-vocabulary word and each prototype, and selects the prototype feature vector with the highest similarity to replace the current output. The updated set of semantic feature vectors is then fed into the graph neural network input.

[0078] Specifically, in the document processing method based on dynamic multi-level indexing and feature clustering described in this invention, step 3 includes:

[0079] The received set of semantic feature vectors is used to define each semantic feature vector as a graph node. The cosine similarity between any two graph nodes is calculated. Undirected edges are created for node pairs with similarity greater than 0.85, generating a relational graph with edge weights.

[0080] The relationship graph is input into a graph convolutional network, and node embedding vectors are generated through a third-order neighbor aggregation operation.

[0081] Perform spectral clustering on the set of node embedding vectors, and output cluster labels and a mapping table between document members and clusters.

[0082] After mapping the semantic feature vector set to graph nodes, the similarity index between any two nodes is calculated. When the similarity exceeds a set threshold, an undirected connection edge is established between the nodes to form an initial graph structure. This graph structure is then input into a graph convolutional network for neighbor feature aggregation, generating node embedding vectors that incorporate neighborhood information through multiple iterations. Spectral clustering analysis is performed on the embedding vector set, dividing it into clusters based on the feature space distribution and outputting two key data points: cluster labels representing type affiliation and a document-to-cluster mapping table. These two data points serve as the foundation for graph construction in step 2.

[0083] Specifically, in the document processing method based on dynamic multi-level indexing and feature clustering described in this invention, step 2 includes: constructing the state space of a reinforcement learning agent, inputting the document update frequency variance, inputting the graph modularity index, and inputting the latency change rate of historical query log statistics;

[0084] The action space is defined as a binary decision space. A binary instruction code of 0 represents a full reconstruction instruction, and a binary instruction code of 1 represents an incremental update instruction. The state space input is processed through a policy network to generate reconstruction probability scores.

[0085] Output instruction code 0 when the probability score exceeds 0.7, otherwise output instruction code 1.

[0086] The binary instruction code is pushed to the index reconstruction executor in real time to trigger the reconstruction operation.

[0087] When constructing the state space of the reinforcement learning agent, three key inputs are integrated: the update frequency variance index, the real-time output graph modularity, and the statistical rate of change of historical query latency. The agent's action space is defined as binary instruction code output, where a specific encoded value represents a full reconstruction instruction, and another encoded value represents an incremental update instruction. A neural network policy module analyzes the state data to generate probability scores. When the score exceeds a decision threshold, a full reconstruction instruction code is triggered; otherwise, an incremental instruction code is triggered. The generated instruction codes are transmitted to the execution module in real time via a message channel.

[0088] Specifically, the document processing method based on dynamic multi-level indexing and feature clustering described in this invention includes the following steps in reconstructing the multi-level index structure: receiving binary instruction codes and parsing the instruction code type.

[0089] If the instruction code is 0, read the cluster center coordinates in the document semantic graph generated in step 2 of claim 1, and map the non-leaf nodes of the B+ tree to the coordinates;

[0090] If the instruction code is 1, locate the current displacement node, obtain its displacement path in the member mapping table of claim 4, and dynamically modify the parent pointer;

[0091] Generate a reconstructed index tree file and register the file to the query routing table in step 4 of claim 1 to support document retrieval.

[0092] Upon receiving the binary instruction code, it is processed according to the encoding type: if it is identified as a full reconstruction instruction code, the spatial coordinates of the cluster centers are read, a mapping relationship between B+ tree index nodes and coordinates is established, and the depth of the index tree hierarchy is compressed; if it is identified as an incremental instruction code, the displacement node path recorded in the member mapping table is located, and its parent node pointer relationship is dynamically adjusted. The reconstructed index tree file is registered to the query routing table in step 4, which provides path guidance for retrieval operations.

[0093] Specifically, in the document processing method based on dynamic multi-level indexing and feature clustering described in this invention, the step of performing locality-sensitive hash matching includes:

[0094] The target document cluster is located by reconstructing the index tree file, and the pre-trained language model is reused as an encoder to generate a locality-sensitive hash function.

[0095] Within the target document cluster, extract the feature vectors of each document member in the cluster, calculate the cosine similarity between each feature vector and the feature vector of the query request, filter document members with a cosine similarity greater than 0.85, and output the filtered document member set as the retrieval result document set to step 5.

[0096] After locating the target document cluster by reconstructing the index tree, a semantic feature extraction model is invoked to generate a locality-sensitive hash function. The similarity index between the query request feature vector and the feature vectors of each document within the cluster is calculated, and document members meeting the matching criteria are selected based on the index value. The selection process employs a dual verification mechanism: first, a coarse screening of potential matches is performed using the hash function, followed by a fine screening using similarity to determine the final result set. The output document member set is then fed into step 5 as feedback for optimization.

[0097] Specifically, in the document processing method based on dynamic multi-level indexing and feature clustering described in this invention, step 5 includes:

[0098] Extract the semantic feature vector of the mismatched document and reuse the query request feature vector as the expected vector;

[0099] Extract the semantic feature vector of the mismatched document and reuse the query request feature vector as the expected vector;

[0100] The Euclidean distance between the feature vector of the mismatched document and the expected vector is calculated as the feature difference value. The feature difference value is then input into the gradient descent optimizer to generate the model parameter adjustment amount.

[0101] The adjustment is backpropagated to the feature extraction model to update the weight parameters, triggering the graph neural network to recalculate the node connections based on the updated parameters.

[0102] The semantic feature vectors of mismatched documents are generated in real time through a feature extraction model. This model uses a pre-trained language architecture to convert text to vectors through context encoding. Simultaneously, the query request feature vectors generated during query request processing are reused as the benchmark for the expected vector. Both types of vectors represent the semantic information of the documents within the same feature space, providing an alignment basis for difference calculation.

[0103] When calculating the spatial distance between the feature vectors of mismatched documents and the expected vectors, the Euclidean distance algorithm is used to quantify their geometric difference. This distance value serves as a scalar measure of feature difference, directly reflecting the degree of bias of the feature extraction model on the current samples. The feature difference value is input into the gradient descent optimizer, and differentiation is used to generate the adjustment direction and magnitude of the model weight parameters.

[0104] The parameter adjustments are backpropagated to the network layers of the feature extraction model, updating the fully connected weights and convolutional kernel parameters layer by layer. After the model weights are updated, a graph neural network reset mechanism is triggered: document node vectors are regenerated based on the updated feature extraction model, and undirected edge connections are reconstructed based on the cosine similarity between nodes. This process simultaneously updates the cluster labels and membership mapping table, achieving a closed-loop feedback from feature optimization to graph reconstruction.

[0105] The entire process forms a self-optimizing technical loop: feature differences drive model parameter updates, the updated model improves feature quality, optimized features improve map accuracy, and accurate maps reduce the probability of subsequent mismatches. Continuous iteration suppresses the impact of noisy documents, reducing invalid index reconstruction operations caused by feature distortion at the source.

[0106] Specifically, the document processing method based on dynamic multi-level indexing and feature clustering described in this invention further includes:

[0107] After the graph neural network outputs cluster labels and member mapping tables, the cluster labels are written to a disk persistent storage file to establish a label library;

[0108] When index reconstruction requires locating cluster center coordinates, the target cluster label is read from the persistent storage file;

[0109] Based on the member document features corresponding to the read cluster labels, a spatial coordinate mapping relationship is constructed through R-tree indexing, and the cluster center coordinates are output to the index reconstruction module.

[0110] After clustering the cluster labels, they are persistently stored as binary files on disk to build a label library. When step 3 requires cluster center mapping, the target cluster label data is loaded from the storage file. Based on the feature set of the member documents corresponding to the labels, a spatial indexing technique is used to construct an accelerated retrieval structure. This index structure outputs the center coordinates to the index reconstruction module, improving coordinate positioning efficiency.

[0111] Specifically, the document processing method based on dynamic multi-level indexing and feature clustering described in this invention is characterized by further comprising:

[0112] After generating the document clustering graph in step 2, the graphics rendering engine is invoked to construct a topological visualization view of the graph;

[0113] After calculating the feature difference values ​​of mismatched documents, extract the feature difference value set of all mismatched documents;

[0114] The set of feature difference values ​​is mapped to the corresponding document node positions in the visualization view to generate a dynamically updated heatmap of difference distribution.

[0115] Based on the document clustering graph generated in step 2, an interactive topology view is constructed using a graphics rendering engine. Simultaneously, a set of feature difference values ​​is acquired and mapped to the corresponding document node positions in the topology view. A heatmap is generated using a color gradient algorithm and overlaid on the topology view to display the spatial distribution of feature differences in real time. This view provides users with a visual diagnostic tool for the system's operational status.

[0116] This invention addresses the overhead of index reconstruction in high-frequency document update environments. A reinforcement learning agent dynamically decides reconstruction strategies based on document update frequency variance and graph modularity. When the reconstruction probability score exceeds a preset threshold, a full reconstruction instruction is output; otherwise, an incremental update instruction is output, avoiding invalid reconstruction operations caused by frequent, minor updates. The index reconstruction execution module responds to instructions with differentiated processing: during full reconstruction, multi-level index nodes are mapped to the cluster center coordinates of the document semantic graph to compress the tree hierarchy depth; during incremental updates, displacement nodes are located through a member mapping table, and parent pointers are dynamically adjusted, only changing the affected subgraphs. The reconstructed index tree file is registered to the query routing table to form a simplified retrieval structure. A closed-loop feedback system drives feature extraction model optimization through mismatch cases, calculates the feature difference between mismatched documents and query requests, backpropagates to generate model parameter gradients, updates the feature extraction model weights, and triggers the graph neural network to recalculate node connections, thereby improving clustering accuracy and suppressing noise interference, ultimately achieving synergistic optimization of index reconstruction overhead and query latency.

[0117] This invention is specifically implemented in high-frequency update scenarios of document management systems, such as financial transaction record systems frequently adding entries or social media platforms inserting user-generated content in real time. The document update event stream captures transaction log streams from the document storage system via a distributed message queue. The transaction log streams are then parsed to generate a structured event stream, which includes triples of document identifiers, timestamps, and the original text. The variance of the number of updates per unit time in the structured event stream is calculated in real time. When the variance exceeds a system-set dynamic threshold, a semantic feature extraction operation is triggered: the original text fields in the structured event stream are extracted, input into a pre-trained language model for context encoding, and a fixed-dimensional dense vector is output as the semantic feature vector. When encountering out-of-vocabulary (OV) words during processing, the prototype document feature vectors in a pre-defined vector library are retrieved, the cosine similarity between the OV word and the prototype document is calculated, and the prototype feature vector with the highest similarity is selected to replace the current output, thus ensuring accurate feature representation.

[0118] When constructing a document semantic graph based on semantic feature vectors, each semantic feature vector is defined as a graph node. The cosine similarity between node pairs is calculated, and for node pairs with similarity exceeding a preset threshold, undirected edges are created to generate a weighted relational graph. The relational graph is input into a graph convolutional network, and node embedding vectors are generated through multi-order neighbor aggregation. A spectral clustering algorithm is then performed on the set of node embedding vectors to output cluster labels and a mapping table between document members and clusters. The graph modularity index is continuously calculated as a stability measure. The document update frequency variance and the graph modularity index are input into the state space of the reinforcement learning agent. Combined with the latency change rate statistically obtained from historical query logs, a reconstruction probability score is generated through a policy network. When the reconstruction probability score exceeds a preset threshold, the reinforcement learning agent outputs the binary instruction code for a full reconstruction instruction; otherwise, it outputs the binary instruction code for an incremental update instruction. This instruction code is pushed to the index reconstruction executor in real time to trigger the operation.

[0119] The index reconstruction execution module handles instruction codes differently: after parsing the instruction code type, if it is identified as a full reconstruction instruction, it reads the cluster center coordinates in the document semantic graph and maps multi-level index nodes to the cluster center coordinates to compress the tree hierarchy depth; if it is identified as an incremental update instruction, it locates the displacement path of the current displacement node in the member mapping table and dynamically adjusts the parent pointer relationship of the displacement node to minimize the reconstruction scope. After reconstruction, an index tree file is generated and registered in the query routing table, forming an optimized retrieval path structure. During the query phase, after receiving an external query request, the module locates the target cluster in the document semantic graph by reconstructing the index tree, reuses the pre-trained language model to generate a locality-sensitive hash function, extracts the feature vectors of document members within the cluster, calculates the similarity index between the feature vectors and the query request feature vectors, and filters document members with similarity exceeding the threshold to output the retrieval result document set.

[0120] For the retrieved document set, the matching confidence score between each document and the query request is calculated, and documents with confidence scores below a threshold are selected as mismatches. Semantic feature vectors of the mismatches and the query request feature vectors are extracted, and the feature difference value is calculated as a Euclidean distance metric. This feature difference value is input into the gradient descent optimizer to generate the model parameter gradient, and the feature extraction model weight parameters are updated through backpropagation. After the weight update, the graph neural network is triggered to recalculate node connections, optimizing the clustering accuracy of the document semantic graph and reducing misassignments caused by noisy documents. The entire implementation process forms a closed-loop feedback system, using dynamic decision-making mechanisms to avoid invalid reconstructions, index structure optimization to reduce local overhead, and continuous model updates to suppress interference, collaboratively solving the problems of index reconstruction overhead and query latency in high-frequency update environments.

[0121] The following is an explanation of the core model and its functions involved in this invention:

[0122] The pre-trained language model is responsible for transforming raw text into machine-understandable semantic features. This model encodes document content into dense vectors of fixed dimensions by analyzing the contextual relationships between words. When encountering a new word not included in the dictionary, the system searches a pre-defined prototype document vector library and finds the prototype feature that is semantically closest to the current word to replace it. This approach enables the text feature extraction process to handle unknown words while maintaining the continuity of feature representation.

[0123] Graph convolutional networks are used to construct semantic relationship graphs between documents. This network treats the feature vector of each document as a graph node, automatically identifying node pairs with similarity exceeding a set threshold and establishing connecting edges. Through a multi-layer neighbor information aggregation mechanism, each node integrates feature information from its neighboring nodes. This network structure effectively captures the community structure of document clusters, providing a topological foundation for subsequent clustering analysis.

[0124] The reinforcement learning agent undertakes the decision-making task for index reconstruction strategies. This agent continuously receives three key metrics: frequency variance reflecting the drasticness of document updates, modularity metric characterizing the stability of the index structure, and historical query latency trends. The agent's internal policy network jointly analyzes these metrics and outputs binary decision instructions. The decision-making mechanism uses probability scores to achieve flexible judgment, avoiding system oscillations caused by frequent switching of reconstruction modes.

[0125] Locality-Sensitive Hash (LSH) matching is used for fast document filtering. This model reuses the structure of a pre-trained language model to generate specific hash functions for document clusters. By mapping high-dimensional feature vectors to a low-dimensional hash space, the model can quickly locate a candidate set of documents similar to the query request. The design of the hash function ensures that similar documents have a higher probability of mapping to the same hash bucket, significantly improving retrieval efficiency.

[0126] The feature extraction model is continuously optimized through closed-loop feedback. When the system detects a mismatch, the model calculates the difference between the actual output and the desired features. Using gradient descent, the model automatically adjusts its internal parameter weights. After parameter updates, the document feature vector is regenerated, thereby altering the node connections in the graph neural network. This closed-loop mechanism enables the system to learn autonomously from erroneous cases, gradually improving the accuracy of feature representation.

[0127] The document update frequency variance algorithm dynamically monitors the intensity of changes in the document library. This algorithm statistically analyzes the dispersion of document update frequency per unit time. When the calculated result exceeds a dynamic threshold, it indicates that the document library has entered a high-frequency update state. This quantification method provides objective evidence for the system to perceive environmental changes, triggering subsequent feature extraction processes.

[0128] The cosine similarity algorithm measures the semantic relevance between documents. The algorithm treats document feature vectors as directions in a multi-dimensional space and determines semantic similarity by calculating the cosine of the angle between the vectors. When the calculation result exceeds a preset threshold, the system determines that the two documents are significantly related. This algorithm provides a relational criterion for constructing semantic graphs.

[0129] Spectral clustering algorithms automatically divide documents into highly cohesive clusters. The algorithm first constructs a document similarity matrix and then identifies dense regions in the vector space through eigenvalue decomposition. Documents are assigned to different clusters based on their vector distribution, and a membership mapping table is generated simultaneously. This process enables the system to autonomously discover the inherent grouping structure of document sets.

[0130] Graph convolutional network neighbor aggregation algorithms enhance the representational power of document features. The algorithm allows each document node to absorb feature information from its neighboring nodes, forming a new vector that incorporates contextual semantics through multiple iterations. Nodes gain richer semantic representations through aggregation operations, providing optimized input data for clustering analysis.

[0131] Locality-Sensitive Hash (LSH) matching algorithm enables fast retrieval of neighboring documents. The algorithm maps high-dimensional feature vectors to a low-dimensional hash space, preserving the similarity relationships in the original space. Similar documents are mapped to the same hash bucket with high probability; the system only needs to compare documents within the same bucket to narrow the search range, significantly improving query efficiency.

[0132] Gradient descent optimization drives the continuous evolution of the feature extraction model. Based on the feature differences of mismatched cases, the algorithm calculates the direction and magnitude of model parameter adjustments. Through backpropagation, network weights are updated layer by layer, causing the model output to gradually approximate the desired features. This mechanism endows the system with adaptive optimization capabilities.

[0133] A binary decision algorithm controls the switching logic of the index reconstruction mode. The algorithm compares the reconstruction probability score with a preset threshold. If the probability exceeds the threshold, it outputs a full reconstruction instruction code; otherwise, it outputs an incremental update instruction code. This binary output mechanism forms a clear reconstruction strategy signal, guiding the index execution module to operate precisely.

Claims

1. A document processing method based on dynamic multi-level indexing and feature clustering, characterized in that, include: Step 1: Obtain the document update event stream, calculate the document update frequency variance in real time, and extract the semantic feature vector of the document in the event stream when the update frequency variance exceeds the dynamic threshold set by the system. Step 2: Construct a document semantic graph based on the semantic feature vectors and continuously output the graph modularity index. Input the update frequency variance and the graph modularity index into the reinforcement learning agent. The reinforcement learning agent makes a decision on the reconstruction strategy and generates a reconstruction probability score. If the reconstruction probability score exceeds a preset threshold, a full reconstruction instruction is output; otherwise, an incremental update instruction is output. Step 2 includes: constructing the state space of the reinforcement learning agent, inputting the document update frequency variance, inputting the graph modularity index, and inputting the latency change rate of historical query log statistics; The action space is defined as a binary decision space. A binary instruction code of 0 represents a full reconstruction instruction, and a binary instruction code of 1 represents an incremental update instruction. The state space input is processed through a policy network to generate reconstruction probability scores. Output instruction code 0 when the probability score exceeds 0.7, otherwise output instruction code 1. The binary instruction code is pushed to the index reconstruction executor in real time to trigger the reconstruction operation; Step 3: Respond to the full reconstruction instruction or the incremental update instruction. When responding to the full reconstruction instruction, map the multi-level index nodes to the cluster center coordinates of the document semantic graph to compress the tree hierarchy. When responding to the incremental update instruction, dynamically adjust the parent pointers of the displacement nodes in the document semantic graph to reduce reconstruction overhead. Step 4: Receive external query requests, locate the target cluster in the document semantic graph, filter all document members included in the cluster, and extract the feature vectors corresponding to each document member. Based on the feature vectors and the query request feature vectors, perform local sensitive hash matching; filter document members according to hash matching similarity, and output the set of filtered document members as the retrieval result document set. Step 5: Calculate the matching confidence of each document in the search results document set with the query request, and filter documents with confidence scores below the threshold as mismatch cases; extract the semantic feature vectors of the mismatch cases and the feature vectors of the query request, and calculate the feature difference value; input the feature difference value into the feature extraction model, generate the model parameter gradient through backpropagation, and update the feature extraction model weights based on the gradient; The process of obtaining the document update event stream includes: Obtain the transaction log stream of the document storage system in real time via message queue; The transaction log stream is parsed to generate a structured event stream, which includes triplet data of document identifier, timestamp, and original text. The variance of the number of updates per unit time in the structured event stream is calculated in real time, and the variance of the number of updates per unit time is used as the basic indicator of the document update frequency variance.

2. The document processing method based on dynamic multi-level indexing and feature clustering according to claim 1, characterized in that, Step 2 includes: Extract the raw text fields from the structured event stream; The original text field is input into a pre-trained language model to perform context encoding, and a 128-dimensional dense vector is output as a semantic feature vector. The model output process dynamically processes out-of-vocabulary words, retrieves prototype document feature vectors from a pre-defined vector library, calculates the cosine similarity between out-of-vocabulary words and prototype documents, selects the prototype feature vector with the highest similarity to replace the current vector, and outputs the updated semantic feature vector as the input to the graph nodes.

3. The document processing method based on dynamic multi-level indexing and feature clustering according to claim 2, characterized in that, Step 3 includes: The received set of semantic feature vectors is used to define each semantic feature vector as a graph node. The cosine similarity between any two graph nodes is calculated. Undirected edges are created for node pairs with similarity greater than 0.85, generating a relational graph with edge weights. The relationship graph is input into a graph convolutional network, and node embedding vectors are generated through a third-order neighbor aggregation operation. Perform spectral clustering on the set of node embedding vectors, and output cluster labels and a mapping table between document members and clusters.

4. The document processing method based on dynamic multi-level indexing and feature clustering according to claim 3, characterized in that, Step 3 includes: Receive binary instruction codes and parse the instruction code type: If the instruction code is 0, read the cluster center coordinates in the document semantic graph generated in step 2 of claim 1, and map the non-leaf nodes of the B+ tree to the coordinates; If the instruction code is 1, locate the current displacement node, obtain its displacement path in the mapping table between the document members and the cluster, and dynamically modify the parent pointer; Generate a reconstructed index tree file and register the file to the query routing table in step 4 of claim 1 to support document retrieval.

5. The document processing method based on dynamic multi-level indexing and feature clustering according to claim 1, characterized in that, Step 4 includes: The target document cluster is located by reconstructing the index tree file, and the pre-trained language model is reused as an encoder to generate a locality-sensitive hash function. Within the target document cluster, extract the feature vectors of each document member in the cluster, calculate the cosine similarity between each feature vector and the feature vector of the query request, filter document members with a cosine similarity greater than 0.85, and output the filtered document member set as the retrieval result document set to step 5.

6. The document processing method based on dynamic multi-level indexing and feature clustering according to claim 5, characterized in that, Step 5 includes: Extract the semantic feature vector of the mismatched document and reuse the query request feature vector as the expected vector; The Euclidean distance between the feature vector of the mismatched document and the expected vector is calculated as the feature difference value. The feature difference value is then input into the gradient descent optimizer to generate the model parameter adjustment amount. The adjustment is backpropagated to the feature extraction model to update the weight parameters, triggering the graph neural network to recalculate the node connections based on the updated parameters.

7. The document processing method based on dynamic multi-level indexing and feature clustering according to claim 1, characterized in that, Also includes: After the graph neural network outputs cluster labels and a mapping table between document members and clusters, the cluster labels are written to a disk persistent storage file to establish a label library. When index reconstruction requires locating cluster center coordinates, the target cluster label is read from the persistent storage file; Based on the member document features corresponding to the read cluster labels, a spatial coordinate mapping relationship is constructed through R-tree indexing, and the cluster center coordinates are output to the index reconstruction module.

8. The document processing method based on dynamic multi-level indexing and feature clustering according to claim 1, characterized in that, Also includes: After generating the document clustering graph in step 2, the graphics rendering engine is invoked to construct a topological visualization view of the graph; After calculating the feature difference values ​​of mismatched documents, extract the feature difference value set of all mismatched documents; The set of feature difference values ​​is mapped to the corresponding document node positions in the visualization view to generate a dynamically updated heatmap of difference distribution.

Images