Structured data storage method and system based on natural language transformation

Abstract

The invention discloses a structured data storage method based on natural language transformation, which comprises the following steps of: a system initialization configuration stage: deploying a protocol adapter in a local memory of a PC (Personal Computer) client, and loading natural language processing pipeline configuration parameters; a heterogeneous data acquisition stage: capturing a multi-source text data stream through the protocol adapter, uniformly converting the multi-source text data stream into a standardized data packet, and sending the standardized data packet to a message queue theme; a text cleaning stage: a named entity recognition stage: inputting the pure text data into an NER module deployed with a language model loader; in the conditional feature extraction stage, feature vectors are generated for texts meeting preset conditions on the basis of entity type tags in the entity recognition result; and a consistent storage stage: inserting the entity identification results into a relational database in batches, and updating the entity mapping relationship in the cache. According to the method, the intelligent level of cache management is remarkably improved, and the access fluency of the key data of the user is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a structured data storage method and system based on natural language conversion. Background Technology

[0002] With the widespread adoption of cloud computing technology, user interaction with cloud storage via client applications has become commonplace. To accelerate file access and save network bandwidth, clients typically employ local hard drive caching. However, local cache space is limited, and when the cache is full, a cache eviction mechanism must be activated to delete some files to make room for new data. Existing technologies, such as Least Recently Used (LRU) and Least Frequently Used (LFU) algorithms, rely solely on file access timestamps and frequency, failing to discern the intrinsic attributes of files. For example, an important file that hasn't been accessed for several days might be prioritized for eviction, while a frequently written but low-value log file might be retained. This "mechanical" decision-making severely impacts cache effectiveness and can even cause user work interruptions at critical moments due to cache misses. Furthermore, while some research attempts to introduce more features and machine learning models to improve evaluation, these models typically require extensive supervised training with labeled data and have limited ability to understand the rich semantic information contained in unstructured text data such as filenames and paths. Summary of the Invention

[0003] The purpose of this invention is to provide a structured data storage method and system based on natural language conversion, thereby solving the aforementioned problems existing in the prior art.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] A structured data storage method based on natural language processing includes the following steps:

[0006] During the system initialization and configuration phase, the protocol adapter is deployed in the local storage of the PC client, the configuration parameters of the natural language processing pipeline are loaded, and the message queue cluster and database connection pool are initialized.

[0007] During the heterogeneous data acquisition phase, multi-source text data streams are captured through a protocol adapter, uniformly converted into standardized data packets, and sent to the message queue topic.

[0008] During the text cleaning stage, data packets are consumed from the message queue, and the text cleaning module is called recursively to remove HTML tags and non-standard encoded characters to generate clean text data.

[0009] In the named entity recognition stage, clean text data is input into the NER module with a language model loader. When the text length exceeds a preset threshold, a segmentation mechanism is triggered. Overlapping segments are segmented using a sliding window strategy. After entity recognition is performed independently on each segment, a confidence-weighted voting mechanism is used to fuse the cross-segment results to generate entity recognition results.

[0010] In the conditional feature extraction stage, based on the entity type label in the entity recognition result, the feature extraction module is triggered to perform TF-IDF vectorization processing to generate feature vectors for text that meet the preset conditions.

[0011] During the consistent storage phase, the target data table is locked through a distributed lock mechanism, and the entity identification results are inserted into the relational database in batches. After the transaction is successfully committed, the primary key identifier is broadcast through the publish-subscribe channel, and the entity mapping relationship in the cache is updated.

[0012] Preferably, the system initialization configuration phase also includes:

[0013] Deploy an event-driven scheduler in a container orchestration environment. The scheduler listens for data readiness events in the message queue and triggers the rule engine. The rule engine defines the dependencies and topology execution order of natural language processing tasks through configuration files.

[0014] Preferably, the named entity recognition stage also includes:

[0015] The GPU scheduler dynamically monitors memory usage. When the usage exceeds a preset warning value, it triggers a gradient checkpoint mechanism. The dynamic batch processing unit adaptively adjusts the size of each inference batch based on the available memory capacity and the average text length.

[0016] Preferably, the named entity recognition stage also includes:

[0017] The entity disambiguation module connects to the knowledge graph API, obtains entity encodings, calculates the similarity of context semantic vectors, and retains the optimal entity sense based on the similarity calculation results.

[0018] Preferably, the consistent storage phase further includes:

[0019] A two-phase commit protocol is used to manage transaction consistency across databases and caches. Before performing batch inserts, a lock is acquired through atomic operations of the cache, and the lock expiration time is dynamically configured according to the business scenario.

[0020] Preferably, it also includes an adaptive optimization phase:

[0021] When the message queue backlog exceeds a preset threshold, the number of processing instances of the named entity recognition module is automatically increased; when a data packet that fails to be processed is identified, it is routed to the dead letter queue and persisted to the fault analysis library.

[0022] A structured data storage system based on natural language processing, deployed on the local storage of a PC client, includes:

[0023] Protocol adapters are used to capture multi-source text data streams and convert them into standardized data packets. The adapters support Hypertext Transfer Protocol, Bidirectional Communication Protocol and Message Queuing Transfer Protocol.

[0024] The data processing layer is asynchronously coupled with the protocol adapter through a message queue cluster. The data processing layer includes a text cleaning module, a named entity recognition module, a feature extraction module, and a semantic parsing module.

[0025] The scheduling and rule engine layer is deployed in a container orchestration environment. The scheduler listens for message queue events and triggers the rule engine to execute the task pipeline defined in the configuration file. The configuration file uses a markup language format to define the task topology and exception policies.

[0026] The storage layer includes a relational database and a cache. The database has data tables corresponding to the data processing output and adopts a time range partitioning strategy. The cache implements a distributed locking mechanism through atomic operations.

[0027] The monitoring management layer is used to monitor message queue consumption progress, processing instance resource utilization, and data consistency status.

[0028] Preferably, the named entity recognition module includes:

[0029] A language model loader that dynamically loads pre-trained models and manages model version caching;

[0030] The segmentation processor uses a sliding window strategy to segment overlapping segments when the text length exceeds a preset threshold.

[0031] The voting fusion unit performs confidence-weighted voting on cross-fragment entity recognition results;

[0032] The disambiguation connector calls the knowledge graph interface to perform entity context connection and sense filtering.

[0033] Preferably, the scheduling and rules engine layer includes:

[0034] Offset manager, uses ordered sets to record message queue consumption progress and supports manual commit;

[0035] An automatic scaling controller dynamically adjusts the number of data processing layer instances based on message backlog metrics.

[0036] The dead-letter queue processor captures failed data and writes it to the fault analysis library. The fault analysis library sets a time-to-live index to automatically clean up expired data.

[0037] Preferably, the storage layer further includes:

[0038] The transaction coordinator uses a distributed transaction framework to manage the atomicity of operations across databases and caches.

[0039] Incremental update triggers automatically refresh the aggregated statistical materialized view when a new record is added to the data table;

[0040] A publish / subscribe notification broadcasts the primary key identifier for downstream services to consume after a transaction is successfully committed.

[0041] The beneficial effects of this invention are:

[0042] This invention utilizes the ability of large language models to understand deep information such as file semantics, path context, and project relevance, enabling accurate prediction of the probability of future file access. This allows for optimal cache retention and eviction decisions, significantly improving cache hit rate, ensuring smooth access to users' critical data, and possessing good interpretability and adaptability to individual user habits.

[0043] This invention elevates the assessment of file access frequency from a single dimension based on "access records" to a comprehensive dimension based on "semantic understanding." The large language model, like a human expert, can deeply analyze the semantic connotations of filenames and directory structures, and perform logical reasoning based on context. This makes its prediction accuracy for the future access probability of files far superior to traditional algorithms, effectively preventing important files from being incorrectly evicted, thereby significantly improving cache hit rate and optimizing user access speed and experience for critical data.

[0044] This invention establishes an implicit user feedback loop, allowing the local model to be continuously fine-tuned so that its decision-making logic gradually aligns with the workflow and file usage habits of specific users. This means the system becomes increasingly adaptable to different roles, ultimately achieving a highly personalized intelligent caching service tailored to each user.

[0045] This invention utilizes "reasoning chain" technology, allowing the model to output the rationale behind its decisions while providing scores. This makes every elimination action in the system transparent, traceable, and auditable, greatly enhancing the system's credibility and maintainability.

[0046] More importantly, since the model and all data processing flows are executed entirely on the user's local machine, no file metadata or user behavior information needs to be uploaded to the cloud, which fundamentally eliminates the risk of data leakage and fully protects the user's privacy and security. Attached Figure Description

[0047] Figure 1 This is a flowchart of the method of the present invention;

[0048] Figure 2This is the system architecture diagram of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0050] Reference Figure 1 The illustrated structured data storage method based on natural language processing includes the following steps:

[0051] S1. During the system initialization and configuration phase, the protocol adapter is deployed in the local storage of the PC client, the configuration parameters of the natural language processing pipeline are loaded, and the message queue cluster and database connection pool are initialized.

[0052] This step specifically involves: Phase S1 aims to build a localized, highly available natural language processing infrastructure, ensuring offline operation throughout the entire process to prevent privacy leaks. The protocol adapter serves as the data entry point, dynamically registering three types of protocol listening services: hypertext transfer, bidirectional communication, and IoT message queues. It automatically parses the data frame structure of heterogeneous data sources through a built-in recognition engine and uniformly converts it into a standardized semantic structure. Pipeline configuration parameters adopt a layered loading strategy: the base layer defines the task topology and timeout thresholds, while the extension layer injects domain-adaptive transfer learning parameters, enabling the model to adapt to new industry terminology without retraining. Message queue cluster initialization requires node discovery, topic creation, and consumer group allocation, configuring persistence and retry mechanisms to achieve asynchronous decoupling between data acquisition and processing. When data volume surges, backpressure flow control is automatically triggered to prevent system overload. The database connection pool dynamically calculates the optimal number of connections based on hardware resources, pre-establishes long connections, and enables health detection to ensure response efficiency in high-frequency write scenarios. The startup of each component follows the dependency inversion principle: first, the storage layer is started to establish a disk persistence channel; then, the message queue is initialized to ensure asynchronous communication; finally, the protocol adapter is loaded to open the entry point, ensuring configuration consistency throughout through atomic operations. This stage provides a resilient, scalable, and fault-isolated operating platform for subsequent processing, capable of handling both single documents and massive data flows, while maintaining the privacy and security advantages of localized processing.

[0053] S2, Heterogeneous data acquisition phase: Multi-source text data streams are captured through a protocol adapter, uniformly converted into standardized data packets, and sent to the message queue topic;

[0054] This step specifically involves: Phase S2, which completes the unified access and standardized flow of heterogeneous data sources. After the protocol adapter starts, it continuously monitors three types of protocol channels: the Hypertext Transfer Protocol interface receives web crawled content, the bidirectional communication protocol port captures real-time session text, and the IoT message queue subscription client collects device log streams. When data arrives, the built-in recognition engine automatically parses the original data frame structure, strips off the protocol header identifier and transmission control information, and extracts the text content from the payload. During parsing, the encoding format is detected synchronously, and non-standard encoded streams are automatically converted and uniformly mapped to a processable character set. Subsequently, metadata such as the source address, timestamp, and protocol type are combined with the text content to construct a standardized semantic structure. This structure contains fixed fields for tracing the data source and collection context. After the structure is generated, it is batch compressed and sent to the designated topic by the message queue producer client, with a reliability mechanism enabled during sending to ensure no packet loss. Topics are pre-divided into different partitions according to business type to achieve physical isolation of data with different priorities. When the queue backlog exceeds a preset threshold, the producer automatically triggers traffic shaping to reduce the collection frequency and avoid system overload. This stage transforms the original raw data streams, which were originally inconsistent in protocol and chaotic in format, into standardized messages with clear structure and explicit semantics, providing a unified input view for subsequent processing. At the same time, the asynchronous buffering mechanism decouples the data acquisition and processing rhythm, enabling the system to withstand instantaneous peak impacts.

[0055] S3, Text Cleaning Stage: Consume data packets from the message queue, call the text cleaning module recursively to remove HTML tags and non-standard encoded characters, and generate clean text data;

[0056] This step specifically involves the S3 phase, which performs deep text cleaning on the original text. The message queue consumer retrieves data packets to be cleaned from the topic in batches, with the number of packets retrieved dynamically adjusted based on the current processing instance load. The retrieval interval is set to a fixed duration to avoid idle processing. Upon receiving the data packets, the text cleaning module first starts the HTML parsing engine, traversing the document object model tree in a top-down manner to identify and remove various tags. For complex webpage structures with deep nesting levels, the parser recursively probes downwards, peeling away container tags and inline style tags layer by layer, preserving the text content and its original order between tags. During parsing, non-standard encoded characters are detected and filtered simultaneously. An automatic identification and replacement mechanism is activated for garbled sequences, mapping unrecognizable characters to placeholders to ensure consistent encoding in the output text. The cleaning module also includes a stop word filter, removing high-frequency function words without actual semantic meaning using a preset vocabulary. After processing, the module generates clean text data that retains the original paragraph structure, sentence boundaries, and key punctuation information, providing high-quality input for subsequent entity recognition. The entire data cleaning process is completed in memory. Upon completion, the heap memory space occupied by the original data packets is immediately released to prevent memory leaks. If a batch of data cleaning fails, the module marks it and moves it to the retry queue. If the retry limit is exceeded, it is moved to the fault isolation area to avoid blocking subsequent batch processing. This stage effectively reduces data noise interference and improves the accuracy and processing efficiency of subsequent analysis stages.

[0057] S4. Named Entity Recognition Stage: Pure text data is input into the NER module with a language model loader. When the text length exceeds the preset threshold, a segmentation mechanism is triggered. Overlapping segments are segmented using a sliding window strategy. After entity recognition is performed independently on each segment, a confidence-weighted voting mechanism is used to fuse the cross-segment results and generate entity recognition results.

[0058] This step specifically involves: In stage S4, deep semantic parsing is performed on the clean text to extract named entities. After receiving the text, the NER module first performs length prediction. When the text size exceeds the single-batch processing capacity limit, a segmentation mechanism is automatically activated. The segmentation processor uses a sliding window strategy to divide the long text into multiple segments, preserving a certain proportion of overlapping areas between adjacent segments to ensure that entity information crossing boundaries is not fragmented. Each segment is independently input into the language model loader. The model encodes and converts the character sequences within the segment, calculates the dependencies between characters through a multi-layer attention mechanism, identifies entities such as names of people, places, and organizations that conform to predefined types, and assigns a confidence score to each recognition result. After segmentation processing, the voting fusion unit initiates cross-segment alignment, performing a confidence-weighted vote on the same entity appearing in overlapping areas. The weight is positively correlated with the confidence value output by the model, and higher confidence results are prioritized in case of boundary conflicts. For entities that appear repeatedly in multiple segments, the system retains the highest confidence version and merges its position information. Finally, a unified entity recognition result is generated, including entity text content, type label, starting position, and confidence level information. This stage combines segmented processing with voting fusion, which not only breaks through the text length limit of a single processing, but also ensures the integrity and accuracy of entity recognition in long texts, effectively avoiding the boundary omission problem caused by traditional methods when processing ultra-long documents.

[0059] S5. Conditional feature extraction stage: Based on the entity type label in the entity recognition result, the feature extraction module is triggered to perform TF-IDF vectorization processing to generate feature vectors for text that meet the preset conditions.

[0060] This step specifically involves the S5 stage, which uses an intelligent feature transformation process triggered by entity type. After receiving the entity recognition results, the feature extraction module first parses the set of entity type labels contained within them, with a pre-defined list of key entity types serving as the trigger condition. When a specific type of entity is detected in the text, the module initiates TF-IDF vectorization processing; if the text contains only general entities or the entity density is below a preset threshold, this stage is skipped to save computational resources. During vectorization, the module performs deep word segmentation, dividing continuous character sequences into semantic units, constructing a vocabulary, and counting the frequency of each unit within the text. It also calculates the weight contribution of each unit based on inverse document frequency. For key phrases in the business scenario, the module enables a multi-phrase capture mode, combining adjacent units to form phrase features. After processing, a sparse feature vector is generated. This vector retains information from high-weight units and filters out low-weight noise terms, achieving an efficient numerical representation of text semantics. This conditional triggering mechanism avoids indiscriminate vectorization of all text, significantly reducing computational overhead while ensuring that important text receives sufficient feature dimensions, improving the accuracy of subsequent classification and retrieval.

[0061] S6. In the consistent storage phase, the target data table is locked through a distributed lock mechanism, and the entity identification results are inserted into the relational database in batches. After the transaction is successfully committed, the primary key identifier is broadcast through the publish-subscribe channel, and the entity mapping relationship in the cache is updated.

[0062] This step specifically involves the S6 phase ensuring that entity identification results are reliably written to disk and synchronized to the cache layer. Before consistent storage begins, the transaction coordinator requests a distributed lock from the cache. The lock key uniquely identifies the target data table. Upon successful acquisition, table-level write permissions are locked to prevent concurrent operations from causing dirty writes. During the lock holding period, the batch inserter groups the entity identification results into pre-defined batch sizes. Each group of data is encapsulated as a transaction unit and submitted to the relational database. The database executes the pre-compiled insert statement and returns confirmation. After successful transaction submission, the transaction coordinator triggers the publish-subscribe channel to broadcast the primary key identifier. The channel pushes the newly added record identifier to all subscribers, notifying them that the data is ready. Simultaneously, the cache updater refreshes the local entity mapping relationship based on the broadcast content, establishing a mapping between the entity summary and the primary key identifier and setting an expiration time. If the transaction rolls back, the distributed lock is automatically released and the temporary mapping in the cache is cleared, ensuring data consistency with the cache state. This phase, through the lock mechanism and publish-subscribe collaboration, achieves atomicity of write operations and real-time cache updates, avoiding query anomalies caused by data inconsistency and improving the overall reliability of the system.

[0063] Preferably, the system initialization configuration phase also includes:

[0064] Deploy an event-driven scheduler in a container orchestration environment. The scheduler listens for data readiness events in the message queue and triggers the rule engine. The rule engine defines the dependencies and topology execution order of natural language processing tasks through configuration files.

[0065] Preferably, the named entity recognition stage also includes:

[0066] The GPU scheduler dynamically monitors memory usage. When the usage exceeds a preset warning value, it triggers a gradient checkpoint mechanism. The dynamic batch processing unit adaptively adjusts the size of each inference batch based on the available memory capacity and the average text length.

[0067] Preferably, the named entity recognition stage also includes:

[0068] The entity disambiguation module connects to the knowledge graph API, obtains entity encodings, calculates the similarity of context semantic vectors, and retains the optimal entity sense based on the similarity calculation results.

[0069] Preferably, the consistent storage phase further includes:

[0070] A two-phase commit protocol is used to manage transaction consistency across databases and caches. Before performing batch inserts, a lock is acquired through atomic operations of the cache, and the lock expiration time is dynamically configured according to the business scenario.

[0071] Preferably, it also includes an adaptive optimization phase:

[0072] When the message queue backlog exceeds a preset threshold, the number of processing instances of the named entity recognition module is automatically increased; when a data packet that fails to be processed is identified, it is routed to the dead letter queue and persisted to the fault analysis library.

[0073] Reference Figure 2 The illustrated structured data storage system based on natural language processing, deployed on the local storage of a PC client, includes:

[0074] Protocol adapters are used to capture multi-source text data streams and convert them into standardized data packets. The adapters support Hypertext Transfer Protocol, Bidirectional Communication Protocol and Message Queuing Transfer Protocol.

[0075] The data processing layer is asynchronously coupled with the protocol adapter through a message queue cluster. The data processing layer includes a text cleaning module, a named entity recognition module, a feature extraction module, and a semantic parsing module.

[0076] The scheduling and rule engine layer is deployed in a container orchestration environment. The scheduler listens for message queue events and triggers the rule engine to execute the task pipeline defined in the configuration file. The configuration file uses a markup language format to define the task topology and exception policies.

[0077] The storage layer includes a relational database and a cache. The database has data tables corresponding to the data processing output and adopts a time range partitioning strategy. The cache implements a distributed locking mechanism through atomic operations.

[0078] The monitoring management layer is used to monitor message queue consumption progress, processing instance resource utilization, and data consistency status.

[0079] Preferably, the named entity recognition module includes:

[0080] A language model loader that dynamically loads pre-trained models and manages model version caching;

[0081] The segmentation processor uses a sliding window strategy to segment overlapping segments when the text length exceeds a preset threshold.

[0082] The voting fusion unit performs confidence-weighted voting on cross-fragment entity recognition results;

[0083] The disambiguation connector calls the knowledge graph interface to perform entity context connection and sense filtering.

[0084] Preferably, the scheduling and rules engine layer includes:

[0085] Offset manager, uses ordered sets to record message queue consumption progress and supports manual commit;

[0086] An automatic scaling controller dynamically adjusts the number of data processing layer instances based on message backlog metrics.

[0087] The dead-letter queue processor captures failed data and writes it to the fault analysis library. The fault analysis library sets a time-to-live index to automatically clean up expired data.

[0088] Preferably, the storage layer further includes:

[0089] The transaction coordinator uses a distributed transaction framework to manage the atomicity of operations across databases and caches.

[0090] Incremental update triggers automatically refresh the aggregated statistical materialized view when a new record is added to the data table;

[0091] A publish / subscribe notification broadcasts the primary key identifier for downstream services to consume after a transaction is successfully committed.

[0092] In this embodiment, the system is deployed on the local storage of the PC client to build a localized, highly available natural language processing infrastructure, ensuring that the entire process runs offline to prevent privacy leaks.

[0093] As a data entry point, the protocol adapter dynamically registers three types of protocol listening services: hypertext transfer, bidirectional communication, and IoT message queues. Through its built-in recognition engine, it automatically parses the data frame structure of heterogeneous data sources, extracts the text content from the payload, and detects the encoding format. It automatically converts non-standard encoded streams, mapping them uniformly to a processable character set. It combines metadata such as source address, timestamp, and protocol type with the text content to construct a standardized semantic structure. This structure is then batch-compressed by the message queue producer client and sent to a designated topic. Topics are pre-partitioned according to business type to achieve physical isolation of data with different priorities. When queue backlog exceeds a preset threshold, the producer automatically triggers traffic shaping to reduce the collection frequency and prevent system overload.

[0094] The data processing layer is asynchronously coupled with the protocol adapter via a message queue cluster. The text cleaning module is configured with a parsing engine and a pre-compiled regular expression filtering unit, recursively removing HTML tags and non-standard encoded characters. The named entity recognition module integrates a language model loader, dynamically loading pre-trained models and managing version caching. When the text length exceeds the processing capacity limit, a segmentation mechanism is automatically activated, using a sliding window strategy to divide the text into multiple segments that retain overlapping regions. Each segment is independently input into the model for encoding conversion and entity recognition. Each recognition result is assigned a confidence score. A voting fusion unit performs confidence-weighted voting on the same entity within the overlapping region, with the weights positively correlated with the model output value. In case of boundary conflicts, higher confidence results are prioritized. The feature extraction module is configured with a vectorization converter, performs deep word segmentation of the text and counts frequencies, calculates weight contributions by combining inverse document frequency, enables a multi-phrase capture mode to form phrase features, and generates sparse feature vectors that retain high-weight information. The semantic parsing module uses an encoder-decoder architecture to convert natural language query statements into a structured representation.

[0095] The scheduling and rules engine layer is deployed in a container orchestration environment. The scheduler listens for message queue data readiness events and triggers the rules engine to execute the task pipeline defined in the configuration file. The configuration file uses a markup language format to define task topology ordering, timeout thresholds, and exception retry strategies. The rules engine uses a DAG builder to convert task dependencies into a directed acyclic graph and generate the topology execution order. It injects domain-adaptive transfer learning parameters based on the business scenario and supports hot switching between A / B testing environments. The offset manager uses ordered sets to record consumption progress and supports manual commits to ensure exact-once processing semantics. The auto-scaling controller dynamically adjusts the number of data processing layer instances based on message backlog metrics. When the number of lagging messages continuously exceeds a threshold, horizontal scaling is triggered. The dead-letter queue processor captures failed processing data and writes it to the fault analysis library. The fault analysis library uses a time-to-live index to automatically clean up expired data, preventing unlimited growth of storage space.

[0096] The storage layer comprises a relational database and a cache. The database is partitioned by monthly time range, and the data tables correspond one-to-one with the output fields of the data processing layer. The cache implements a distributed locking mechanism through atomic operations. The transaction coordinator uses a distributed transaction framework to manage the atomicity of operations across databases and the cache. After a global transaction is initiated, the target data table is locked through the cache. The batch inserter submits the results in batches. After a transaction is successfully committed, the publish / subscribe notification broadcasts the primary key identifier. The cache updater refreshes the local entity mapping relationship and sets the expiration time based on the broadcast content. Incremental update triggers automatically refresh the aggregated statistical materialized view when a new record is added to the data table, using a fast refresh mode to implement incremental updates. The monitoring management layer monitors the message queue consumption progress, processing instance resource utilization, and data consistency status in real time. It collects processing latency and throughput metrics for each module through instrumentation. When an anomaly is detected, alarms are triggered and audit logs are recorded.

[0097] This system automates the entire natural language processing process while ensuring privacy and security through localized deployment and a distributed collaboration mechanism. It features elastic scalability, fault isolation, and high reliability.

[0098] Example 1: Localized Intelligent Customer Service Dialogue Analysis System

[0099] This embodiment applies to PC-based intelligent customer service software, with the system deployed on the local storage of the customer service agent's workstation. When customer service representatives communicate with customers via instant messaging tools, the protocol adapter captures the dialogue text stream in real time, encapsulates each message along with a timestamp and customer identifier into a standardized structure, and pushes it to the message queue. The text cleaning module consumes the queue data, recursively stripping HTML format tags, special symbols, and emoticons from the dialogue, outputting clean dialogue content. The named entity recognition module loads a pre-trained customer service domain model. When the number of dialogue rounds exceeds the length limit, it automatically segments the dialogue into overlapping segments for separate recognition. For customer names and order numbers appearing across segments, a confidence-weighted voting method is used to determine the final boundary, eliminating recognition breaks caused by segmentation. When the feature extraction module detects that an entity contains key types such as product model or fault symptoms, it triggers TF-IDF vectorization to generate a dialogue summary vector. The storage layer locks the daily dialogue table using a distributed lock, writes the identified entities and summaries in batches, and broadcasts the primary key identifier after the transaction is committed, synchronously updating the customer profile mapping in the cache. The monitoring management team tracks queue backlog in real time. When message delays exceed a threshold during peak conversation periods, the automatic scaling controller activates backup processing instances for parallel consumption, ensuring real-time conversation analysis. The entire process runs locally in a closed loop, with conversation content not uploaded to the cloud, protecting enterprise customer data privacy. Asynchronous processing and dynamic scaling support concurrent processing needs during peak periods.

[0100] Example 2: Long Text Legal Document Processing System

[0101] This embodiment addresses the processing of long text documents such as judgments and contracts in law firms. The protocol adapter monitors a designated folder, automatically capturing the file content and sending it to a message queue when a lawyer uploads a new legal document. The text cleaning module removes non-text elements such as case numbers and official seal images while preserving the paragraph structure. The named entity recognition module, handling contract texts that can easily reach hundreds of pages, employs a sliding window segmentation strategy, retaining overlapping areas within each paragraph and independently identifying legal entities such as parties, subject matter, and amounts. A voting fusion unit weights the confidence of the same party's name appearing across pages, prioritizing recognition results with higher confidence. The disambiguation connector interfaces with the legal knowledge graph, performing contextual association on referential entities such as "Party A" and "Party B" to complete their full names. The feature extraction module generates feature vectors only for paragraphs containing key entity types such as points of contention and breach of contract clauses, avoiding computational waste from full-text vectorization. The storage layer stores entity recognition results in partitions by case number, incrementally updating the materialized view of the case knowledge graph after each transaction commit. When lawyers upload multiple documents in batches, the automatic scaling controller adds processing instances in real time based on the queue depth, ensuring that batch tasks are completed within minutes. The system runs entirely locally, meeting the legal profession's requirements for the confidentiality of case information.

[0102] Example 3: Real-time Analysis System for Industrial Equipment Logs

[0103] This embodiment is applied to an edge computing gateway in a factory workshop, processing log data from devices such as PLCs and sensors. The protocol adapter subscribes to device log topics via an IoT message queue protocol, receiving log entries containing timestamps, device numbers, and status codes in real time. A text cleaning module filters hexadecimal dump data from the logs, retaining only textual descriptions. A named entity recognition module identifies entities such as device model, fault code, and alarm level based on the characteristics of the device logs. When a single log entry exceeds a threshold, it is segmented by a fixed step size, and a voting fusion unit merges device number entities across segments. A feature extraction module generates TF-IDF vectors for logs containing fault codes, used for subsequent fault mode classification. The storage layer locks the daily log table using a distributed lock, writes the recognition results in batches, and after the transaction is committed, the publish-subscribe channel pushes the primary key identifier to the fault warning service, triggering real-time alarms. The monitoring management layer monitors message queue backlogs. When a large-scale production line failure causes a log surge, it automatically expands the processing instance and starts a dead-letter queue to isolate abnormal logs, preventing system crashes. The system is deployed on the gateway's local storage, ensuring that log data does not leave the factory and meets industrial data security standards.

[0104] Example 4: Adaptive Learning and Optimization Mechanism

[0105] After the system accumulated a certain amount of processing logs, the monitoring and management team analyzed the failed cases in the dead-letter queue and found that the NER recognition accuracy was declining due to new industry terminology. The adaptive optimization module automatically collected these failed samples, injected them into the domain-adaptive training loop, froze the parameters of the first few layers of the model, and only fine-tuned the last few layers and the top classifier. The learning rate was set to a low level, and mixed precision training was used to reduce memory usage. The fine-tuned new version of the model was hot-swapped and deployed through the version control unit of the rule engine without downtime. At the same time, the automatic scaling controller predicted business peaks based on historical load patterns and expanded the processing instances in advance to achieve resource pre-scheduling. The entity mapping relationships in the cache automatically evict low-frequency entries based on access frequency, freeing up space for new entities. This adaptive mechanism enabled the system to continuously optimize performance as business evolved without manual intervention. After running for several months, the recognition accuracy steadily improved, and the processing latency remained stable, demonstrating a high degree of autonomous evolution capability.

[0106] Example 5: Implementation of Multi-Tenant Isolation and Privacy Protection

[0107] This system supports providing isolated natural language processing services for multiple user accounts on a single PC. The protocol adapter assigns each user an independent protocol listening port and message queue topic prefix. The data processing layer loads the corresponding user's configuration parameters and model version upon startup. The storage layer creates an independent database schema and cache namespace for each user. The distributed lock key name includes the user identifier, ensuring that data write operations for different users do not interfere with each other. The transaction coordinator uses a two-phase commit protocol to guarantee the consistency of user data and cache state. The monitoring and management layer maintains an independent performance indicator dashboard for each tenant, allocating processing instances according to tenant weight when resources are scarce. The entire system is deployed on the PC's local storage, with physical isolation between tenant data. Data processing for any user does not pass through globally shared components, eliminating the risk of cross-user data leakage and meeting the privacy compliance requirements of highly sensitive scenarios such as finance and healthcare.

[0108] By adopting the above-disclosed technical solution of this invention, the following beneficial effects are obtained:

[0109] This invention achieves privacy, security, and high-efficiency operation across the entire natural language processing workflow through localized deployment and a distributed architecture. The system is deployed on the local storage of the PC client, with all data processing completed offline, fundamentally eliminating the risk of leakage caused by uploading sensitive text to the cloud and meeting the high compliance requirements of scenarios such as finance and healthcare. Asynchronous coupling of message queues and event-driven scheduling mechanisms enable the system to have elastic scalability, dynamically responding to data surges and avoiding processing bottlenecks. The named entity recognition module employs segmented processing and confidence-weighted voting fusion, significantly improving the accuracy of long text recognition and eliminating boundary omissions. Conditional feature extraction and batch storage strategies greatly reduce computation and I / O overhead, improving resource utilization. Distributed locks and a two-phase commit protocol ensure cross-component data consistency; after transaction commit, real-time cache synchronization via publish-subscribe ensures accurate and reliable query results. The adaptive optimization module scales up and down in advance based on load prediction, continuously evolving with model fine-tuning, making the system increasingly intelligent with use. The monitoring management layer provides end-to-end observability, and fault isolation and compensation mechanisms ensure high service availability. Overall, this invention achieves automation, intelligence, and high performance in text processing while maintaining privacy, and is suitable for diverse application scenarios that require local offline processing and have stringent requirements for accuracy and consistency.

[0110] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A structured data storage method based on natural language processing, characterized in that, Includes the following steps: During the system initialization and configuration phase, the protocol adapter is deployed in the local storage of the PC client, the configuration parameters of the natural language processing pipeline are loaded, and the message queue cluster and database connection pool are initialized. During the heterogeneous data acquisition phase, the protocol adapter captures multi-source text data streams, converts them into standardized data packets, and sends them to the message queue topic. During the text cleaning stage, data packets are consumed from the message queue, and the text cleaning module is called to recursively remove HTML tags and non-standard encoded characters to generate clean text data. In the named entity recognition stage, the clean text data is input into the NER module with a language model loader. When the text length exceeds a preset threshold, a segmentation mechanism is triggered. Overlapping segments are segmented by a sliding window strategy. After entity recognition is performed independently on each segment, a confidence-weighted voting mechanism is used to fuse the cross-segment results to generate entity recognition results. In the conditional feature extraction stage, based on the entity type label in the entity recognition result, the feature extraction module is triggered to perform TF-IDF vectorization processing to generate feature vectors for text that meet preset conditions. During the consistent storage phase, the target data table is locked through a distributed lock mechanism, and the entity identification results are inserted into the relational database in batches. After the transaction is successfully committed, the primary key identifier is broadcast through the publish-subscribe channel, and the entity mapping relationship in the cache is updated.

2. The method according to claim 1, characterized in that, The system initialization and configuration phase also includes: Deploy an event-driven scheduler in a container orchestration environment. The scheduler listens for data readiness events in the message queue and triggers a rules engine. The rules engine defines the dependencies and topology execution order of natural language processing tasks through a configuration file.

3. The method according to claim 1, characterized in that, The named entity recognition stage also includes: The GPU scheduler dynamically monitors the video memory usage rate. When the usage rate exceeds a preset warning value, it triggers a gradient checkpoint mechanism. The dynamic batch processing unit adaptively adjusts the size of a single inference batch based on the free video memory capacity and the average text length.

4. The method according to claim 1, characterized in that, The named entity recognition stage also includes: The entity disambiguation module connects to the knowledge graph API, obtains entity encodings, calculates the similarity of context semantic vectors, and retains the optimal entity sense based on the similarity calculation results.

5. The method according to claim 1, characterized in that, The consistent storage phase also includes: A two-phase commit protocol is used to manage transaction consistency across databases and caches. Before performing batch inserts, a lock is acquired through atomic operations of the cache, and the lock expiration time is dynamically configured according to the business scenario.

6. The method according to claim 1, characterized in that, It also includes an adaptive optimization phase: When the message queue backlog exceeds a preset threshold, the number of processing instances of the named entity recognition module is automatically increased; when a data packet that fails to be processed is identified, it is routed to the dead letter queue and persisted to the fault analysis library.

7. A structured data storage system based on natural language processing, deployed on the local storage of a PC client, characterized in that, include: A protocol adapter is used to capture multi-source text data streams and convert them into standardized data packets. The adapter supports Hypertext Transfer Protocol, Bidirectional Communication Protocol and Message Queuing Transfer Protocol. The data processing layer is asynchronously coupled with the protocol adapter through a message queue cluster. The data processing layer includes a text cleaning module, a named entity recognition module, a feature extraction module, and a semantic parsing module. The scheduling and rule engine layer is deployed in a container orchestration environment. The scheduler listens to message queue events and triggers the rule engine to execute the task pipeline defined in the configuration file. The configuration file uses a markup language format to define the task topology and exception policies. The storage layer includes a relational database and a cache. The database has data tables corresponding to the data processing output and adopts a time range partitioning strategy. The cache implements a distributed locking mechanism through atomic operations. The monitoring management layer is used to monitor message queue consumption progress, processing instance resource utilization, and data consistency status.

8. The system according to claim 7, characterized in that, The named entity recognition module includes: A language model loader that dynamically loads pre-trained models and manages model version caching; The segmentation processor uses a sliding window strategy to segment overlapping segments when the text length exceeds a preset threshold. The voting fusion unit performs confidence-weighted voting on cross-fragment entity recognition results; The disambiguation connector calls the knowledge graph interface to perform entity context connection and sense filtering.

9. The system according to claim 7, characterized in that, The scheduling and rule engine layer includes: Offset manager, uses ordered sets to record message queue consumption progress and supports manual commit; An automatic scaling controller dynamically adjusts the number of data processing layer instances based on message backlog metrics. The dead-letter queue processor captures and writes failed data into a fault analysis library, which is configured with a time-to-live index to automatically clean up expired data.

10. The system according to claim 7, characterized in that, The storage layer also includes: The transaction coordinator uses a distributed transaction framework to manage the atomicity of operations across databases and caches. Incremental update triggers automatically refresh the aggregated statistical materialized view when a new record is added to the data table; A publish / subscribe notification broadcasts the primary key identifier for downstream services to consume after a transaction is successfully committed.

Images