Dynamic business data updating and querying method and system

Abstract

The invention discloses a dynamic business data updating and querying method and system, and particularly relates to the technical field of information processing, and the method comprises the following steps: S1, real-time data acquisition and preprocessing; s2, updating the dynamic data; s3, intelligent query optimization; s4, result feedback and visualization are carried out; and S5, performing anomaly detection and self-repairing. According to the method, the real-time processing capability and query efficiency of massive heterogeneous data are remarkably improved, the distributed sensor network is combined with lightweight encryption and a differential privacy mechanism to realize localized cleaning and feature extraction of multi-source data, the load of a central server is effectively reduced, and the risk of sensitive information leakage is avoided; an incremental synchronization mechanism depends on a message queue and Hash chain log technology to ensure the atomicity and traceability of data updating in a high-concurrency scene, and by combining a dynamically generated space-time multi-dimensional index and an adaptive weight distribution algorithm, the system can intelligently analyze query semantics and optimize path selection, and the response time of complex query is greatly shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of information processing, and particularly relates to a method and system for dynamic business data update and query. Background Art

[0002] The dynamic business data update and query technology aims to solve the enterprise's requirements for efficient management and agile response to massive, multi-source heterogeneous data in real-time scenarios. Traditional solutions are mostly based on batch processing architectures and relational databases, and achieve data storage and retrieval through full-scale data synchronization and static indexing strategies, which are suitable for low-frequency and structurally stable business scenarios such as report generation or offline analysis; its core processes include periodic data import, predefined query optimization, and result display with fixed granularity. Such technologies once provided basic support for early enterprise informatization;

[0003] However, in the face of high-concurrency updates (such as real-time transactions and Internet of Things device stream data), dynamic query requirements (such as multi-dimensional correlation analysis), and edge computing environments, limitations gradually emerge, and traditional solutions have problems such as high-concurrency latency, low static indexing efficiency, insufficient edge privacy protection, poor architecture scalability, and visualization rigidity. These technical shortcomings severely restrict the enterprise's capabilities in real-time decision-making, resource optimization, and risk prevention and control. Summary of the Invention

[0004] The main purpose of the present invention is to provide a method and system for dynamic business data update and query, which can effectively solve the problems of high-concurrency latency processing and lack of data security.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] A method for dynamic business data update and query, comprising the following steps:

[0007] S1: Real-time data collection and preprocessing: Collect multi-source heterogeneous business data through a distributed sensor network, and perform data cleaning, normalization, and feature extraction based on preset rules;

[0008] S2: Dynamic data update: Adopt an incremental synchronization mechanism, combined with timestamp and version control technologies, to perform real-time update on the business data in the database to ensure data consistency;

[0009] S3: Intelligent query optimization: Dynamically generate multi-dimensional indexes based on the semantic features and load status of the query request, and optimize the query path using an adaptive weight allocation algorithm;

[0010] S4: Result feedback and visualization: Generate a visualization view of the query result through a dynamic rendering engine, and adjust the display granularity in real time according to user interaction behavior;

[0011] S5: Anomaly Detection and Self-Healing: Real-time monitor data stream anomalies through a pre-trained deep learning model, and trigger a data rollback or redundant node replacement mechanism to ensure system stability.

[0012] Preferably, the distributed sensor network in S1 uses edge computing nodes for local preprocessing, transmits data to the central server through a lightweight encryption protocol, and introduces differential privacy technology in the feature extraction stage to protect sensitive data information.

[0013] Preferably, the incremental synchronization mechanism in S2 realizes asynchronous communication through a message queue middleware, and the version control technology adopts an immutable log recording method based on a hash chain to ensure the traceability and anti-tampering of data updates.

[0014] Preferably, the adaptive weight allocation algorithm in S3 dynamically adjusts the index weights according to the historical query response time, data distribution density, and node load status, and preferentially selects query paths with low latency and high availability.

[0015] Preferably, the deep learning model in S5 adopts a hybrid architecture of a temporal convolutional neural network (TCN) and a long short-term memory network (LSTM), analyzes the time series characteristics of the data stream in real time, and starts the self-healing process through a threshold trigger mechanism.

[0016] A system for dynamic business data update and query, the system for dynamic business data update and query is configured to:

[0017] Data acquisition module: Real-time collect multi-source heterogeneous business data through a distributed sensor network, and perform data cleaning, encryption, and feature extraction;

[0018] Dynamic update module: Based on the incremental synchronization mechanism and version control technology, realize the real-time update and consistency maintenance of business data;

[0019] Query optimization module: Generate multi-dimensional indexes according to query semantic features, and optimize query paths using an adaptive algorithm;

[0020] Visualization engine module: Dynamically render query results into interactive visualization views, supporting users to customize the display granularity;

[0021] Anomaly handling module: Real-time monitor system anomalies through a deep learning model, and trigger a self-healing mechanism to ensure data integrity and service continuity.

[0022] Preferably, the dynamic update module integrates a message queue middleware and a hash chain log technology, supporting asynchronous data synchronization and anti-tampering audit functions in high-concurrency scenarios.

[0023] Preferably, the visualization engine module is built with a dynamic rendering algorithm, supports cross-platform and multi-terminal adaptation, and adjusts the spatio-temporal resolution of data display in real time according to user interaction behaviors.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. By integrating edge computing, incremental synchronization, and intelligent optimization technologies, the present invention significantly improves the real-time processing ability and query efficiency of massive heterogeneous data; the distributed sensor network combines lightweight encryption and differential privacy mechanisms to realize the local cleaning and feature extraction of multi-source data while ensuring the secure transmission of data, effectively reducing the load of the central server and avoiding the risk of sensitive information leakage; the incremental synchronization mechanism relies on message queues and hash chain log technologies to ensure the atomicity and traceability of data updates in high-concurrency scenarios. Combined with the dynamically generated spatio-temporal multi-dimensional index and adaptive weight allocation algorithm, the system can intelligently parse query semantics and optimize path selection, greatly shortening the response time of complex queries and adapting to the dynamic change requirements of business scenarios.

[0026] 2. In terms of exception handling and system robustness, the TCN-LSTM hybrid model realizes high-precision real-time monitoring and self-healing capabilities by capturing the temporal characteristics and abnormal patterns of data streams; when data anomalies or node failures are detected, the system automatically triggers data rollback or redundant node replacement mechanisms, combined with the tamper-proof logs maintained by blockchain technology, to ensure business continuity and data integrity. At the same time, the dynamic rendering engine converts complex query results into intuitive views through interactive visualization and multi-granularity display, supports users to adjust the spatio-temporal resolution in real time, and significantly improves the flexibility and decision-making efficiency of data analysis.

[0027] 3. From the embedded edge nodes to the cloud distributed database, and from the micro-frontend visualization components to the containerized operation and maintenance system, each module of the present invention operates in coordination through standardized interfaces, taking into account high scalability and easy maintainability; this end-to-end technology integration not only solves the deficiencies of traditional static data management methods in terms of real-time performance, security, and flexibility, but also provides reliable technical support for the dynamic business needs in the fields of intelligent manufacturing, smart cities, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is the flow of the present invention Figure 1 ;

[0029] Figure 2 is the flow of the present invention Figure 2 . DETAILED DESCRIPTION OF THE INVENTION

[0030] To make the technical means, creative features, achieved objectives and effects of the present invention easy to understand, the present invention will be further described below in conjunction with specific embodiments.

[0031] Example 1, as Figure 1-2 shown, a method for dynamic service data update and query includes the following steps:

[0032] S1: Real-time data collection and preprocessing: Collect multi-source heterogeneous service data through a distributed sensor network, and clean, normalize and extract features from the data based on preset rules;

[0033] The distributed sensor network is deployed using a hierarchical architecture. Edge computing nodes (such as ARM architecture embedded devices) are deployed near the data source end, responsible for preliminary local data cleaning (such as removing duplicate values and filling missing values) and feature extraction (based on predefined feature engineering templates); Data normalization uses dynamic range scaling technology to adaptively adjust the numerical range according to the sensor type. In the feature extraction stage, a lightweight differential privacy algorithm (such as Laplace noise injection) is introduced, and the risk of sensitive information leakage is controlled through the privacy budget (e). The cleaned data is encrypted and transmitted to the central server through the MQTT protocol, and the AES-256 algorithm is used in the encryption process to ensure the security of the transmission link;

[0034] S2: Dynamic data update: Adopt an incremental synchronization mechanism, combine timestamp and version control technology to perform real-time update on the service data in the database to ensure data consistency;

[0035] The incremental synchronization is implemented based on the Apache Kafka message queue. The data update request is encapsulated as an event message and published to a specific topic. After the consumer service subscribes to the message, the incremental data is written into a distributed database (such as Cassandra) in timestamp order. The version control uses the hash chain technology. Each update generates a unique hash value (SHA-3 algorithm) and forms an immutable log with the previous hash link; for example, the hash value of the nth update is calculated by Hash(n) = Hash(Hash(n - 1) || current data block) to ensure the integrity and traceability of the historical record;

[0036] S3: Intelligent query optimization: Dynamically generate multi-dimensional indexes based on the semantic features and load status of the query request, and optimize the query path using an adaptive weight allocation algorithm;

[0037] The query semantic parsing uses an NLP model (such as a fine-tuned version of BERT) to extract keywords and intents. For example, it identifies the time range and aggregation function corresponding to "real-time inventory statistics"; the multi-dimensional index dynamic generation rule is as follows: for high-frequency query fields (such as timestamps, geographical locations), a B+ tree index is automatically created, and for low-frequency fields, a bitmap index is used; the adaptive weight allocation algorithm is based on a reinforcement learning framework, which collects the load of each node (CPU, memory, network latency) and historical query response times in real time, and dynamically adjusts the index weights through a Q-Learning model, and preferentially routes to nodes with low load and fast response;

[0038] S4: Result feedback and visualization: Generate a visualization view of the query result through a dynamic rendering engine, and adjust the display granularity in real time according to the user interaction behavior;

[0039] The dynamic rendering engine is implemented based on WebGL and the D3js library, supporting the switching of multiple views such as heat maps and time series curves; user interaction behaviors (such as zooming and dragging) are fed back to the server in real time through WebSocket, triggering the adjustment of data granularity; for example, when the zoom level of the map view changes, the backend dynamically aggregates geographical data according to the current resolution, reducing the amount of data transmitted; the display granularity adjustment algorithm uses an R-tree spatial index to quickly filter out the data subset that meets the current view range;

[0040] S5: Anomaly detection and self-healing: Real-time monitor data stream anomalies through a pre-trained deep learning model, and trigger a data rollback or redundant node replacement mechanism to ensure system stability.

[0041] The anomaly detection model uses a TCN-LSTM hybrid architecture: the TCN layer (dilated causal convolution) captures the long-term dependencies of the data stream, and the LSTM layer identifies short-term fluctuation patterns; the training data includes normal business data injected with simulated anomalies (such as sudden increases, cliff-like drops), and the model outputs an anomaly probability score; when the score exceeds a dynamic threshold (automatically adjusted according to the recent data volatility), the self-healing process is triggered: if it is a data anomaly (such as sensor drift), roll back to the most recent stable version; if it is a node failure, replace the redundant node through a Kubernetes API call and reallocate the data shards.

[0042] The distributed sensor network in S1 uses edge computing nodes for local preprocessing, transmits data to the central server through a lightweight encryption protocol, and introduces differential privacy technology in the feature extraction stage to protect data sensitive information;

[0043] Specifically, the local preprocessing process of the edge computing node includes: adaptive adjustment of the data sampling rate (dynamically reducing the sampling frequency according to the network bandwidth to reduce the transmission volume), outlier filtering based on the rule engine (such as removing data beyond the 3σ range); implementation details of differential privacy: adding Laplace noise to numerical features (the noise volume is △f / ε, where △f is the feature sensitivity), and adopting the random response mechanism for categorical features (such as randomly replacing the sensitive field "user ID" with a virtual ID with probability p);

[0044] The encryption protocol selects DTLS1.3, and the pre-shared key (PSK) is adopted in the handshake stage to accelerate the negotiation process, which is suitable for low-power sensor devices.

[0045] The incremental synchronization mechanism in S2 realizes asynchronous communication through the message queue middleware, and the version control technology adopts the non-tamperable log recording method based on the hash chain to ensure the traceability and anti-tampering of data updates.

[0046] Furthermore, the partitioning strategy of the Kafka message queue is allocated by hashing according to the data source ID to ensure the order of data from the same source; the consumer group consumes messages with the "at least once" semantics to avoid data loss;

[0047] The hash chain log is stored in the IPFS distributed storage, and each block contains a timestamp, an operation type (add / delete / modify), and a data fingerprint;

[0048] The anti-tampering verification is realized through the Merkle tree: any data modification will cause the root hash value to change, and the tampering location can be quickly located during the audit.

[0049] The adaptive weight allocation algorithm in S3 dynamically adjusts the index weight according to the historical query response time, data distribution density, and node load status, and preferentially selects the query path with low latency and high availability.

[0050] Specifically, the weight calculation formula is: where R i is the historical average response time of node i, D i is the data distribution density (the proportion of the target data stored in this node), L i is the current load rate (normalized value from 0 to 1), and α, β, and γ are adjustable parameters; the algorithm pulls the metric data from the Prometheus monitoring system every 5 seconds and updates the weight; the query router (such as the Envoy proxy) distributes requests according to the weight probabilistically to avoid overloading a single node.

[0051] The deep learning model in S5 adopts a hybrid architecture of the temporal convolutional neural network (TCN) and the long short-term memory network (LSTM) to analyze the time series features of the data stream in real time, and starts the self-repair process through the threshold trigger mechanism.

[0052] Furthermore, the input of the TCN-LSTM model is the standardized data stream within a sliding window (such as 60 seconds). The TCN layer is configured as follows: the convolutional kernel size is 3, the dilation factor is [1, 2, 4], and the number of channels is 64; the LSTM layer is a bidirectional structure with 128 hidden units; the output of the model is converted into an anomaly probability through the Sigmoid function. The initial value of the threshold is 0.7 and it is dynamically adjusted according to the F1-score within the last hour (for example, when the F1 score drops, the threshold is lowered to improve the recall rate); the self-healing process is orchestrated by a workflow engine (such as Apache Airflow), which supports manual intervention confirmation to avoid false triggering.

[0053] A system for dynamic business data update and query, the system for dynamic business data update and query is configured as follows:

[0054] Data acquisition module: Real-time collection of multi-source heterogeneous business data through a distributed sensor network, and perform data cleaning, encryption, and feature extraction;

[0055] The hardware of the data acquisition module consists of a Raspberry Pi edge gateway and LoRa sensor nodes. The software stack includes: an Apache NiFi-based data pipeline (defining cleaning rules and encryption processes), a TensorFlow Lite inference engine (running a feature extraction model); the module supports plug-in expansion. For example, when adding a new sensor type, only the corresponding driver plug-in and feature template need to be loaded;

[0056] Dynamic update module: Based on the incremental synchronization mechanism and version control technology, realize the real-time update and consistency maintenance of business data;

[0057] The core component of the dynamic update module is the Kafka Connect connector, which converts the database change data capture (CDC) into Kafka messages; the version control service is implemented based on the blockchain framework Hyperledger Fabric. Each participating node (such as a data center, an auditing party) jointly maintains a hash chain ledger to ensure global consistency;

[0058] Query optimization module: Generate multi-dimensional indexes according to query semantic features, and optimize the query path using an adaptive algorithm;

[0059] The query optimization module integrates the Apache Calcite framework to parse SQL semantics. The index generator automatically selects column storage (Parquet) or row storage (Avro) according to the query pattern; the adaptive algorithm is deployed as a Kubernetes Operator, which monitors cluster resource metrics and dynamically adjusts the index distribution;

[0060] Visualization Engine Module: Dynamically renders query results into interactive visualization views, supporting users to customize the display granularity;

[0061] The visualization engine adopts a micro-frontend architecture, and each view component (such as maps and dashboards) is developed independently and loaded dynamically; Rendering performance optimization includes: WebWorker offline computing and Canvas off-screen rendering;

[0062] Exception Handling Module: Real-time monitors system exceptions through a deep learning model and triggers a self-healing mechanism to ensure data integrity and service continuity.

[0063] The exception handling module relies on the Flink stream processing engine to calculate data stream features in real time. The model inference service is deployed through the Triton InferenceServer and supports GPU acceleration; The self-healing API is integrated with the operation and maintenance system (such as Ansible) and supports predefined repair scripts (such as restarting services and switching database replicas).

[0064] The Dynamic Update Module integrates message queue middleware and hash chain log technology, supporting asynchronous data synchronization and anti-tampering auditing functions in high-concurrency scenarios.

[0065] Specifically, the message queue adopts a partitioned horizontal expansion design, and the throughput of a single partition can reach 100,000 messages per second; The hash chain log realizes automatic auditing through smart contracts: when a hash mismatch is detected, the contract automatically freezes the suspicious node and notifies the administrator;

[0066] The auditing interface provides a RESTful API, supporting querying logs by time range or operation type.

[0067] The Visualization Engine Module has a built-in dynamic rendering algorithm, supporting cross-platform and multi-terminal adaptation, and adjusting the spatio-temporal resolution of data display in real time according to user interaction behaviors.

[0068] Furthermore, the cross-platform adaptation is based on the ReactNative framework, and the core rendering logic is encapsulated as an independent SDK; The spatio-temporal resolution adjustment algorithm adopts quadtree (geographical data) and B+ tree (time series data) indexes, and dynamically loads data blocks at different precision levels; For example, in a low-network-speed environment on mobile devices, it automatically degrades to hourly granularity, while on the PC side, it supports second-level refreshing.

[0069] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and what is described in the above embodiments and the specification is only to illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will also have various changes and improvements, and these changes and improvements fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for dynamic business data update and query, comprising the following steps: S1: Real-time data collection and preprocessing: Collect multi-source heterogeneous business data through a distributed sensor network, and clean, normalize, and extract features from the data based on preset rules; S2: Dynamic data update: Adopt an incremental synchronization mechanism, combine timestamp and version control technologies, and perform real-time update on the business data in the database to ensure data consistency; S3: Intelligent query optimization: Dynamically generate multi-dimensional indexes based on the semantic features and load status of query requests, and optimize query paths using an adaptive weight allocation algorithm; S4: Result feedback and visualization: Generate a visualization view of the query result through a dynamic rendering engine, and adjust the display granularity in real time according to user interaction behaviors; S5: Anomaly detection and self-repair: Real-time monitor data stream anomalies through a pre-trained deep learning model, and trigger a data rollback or redundant node replacement mechanism to ensure system stability.

2. The method for updating and querying dynamic service data according to claim 1, wherein: In the distributed sensor network in S1, edge computing nodes are used for local preprocessing, data is transmitted to the central server through a lightweight encryption protocol, and differential privacy technology is introduced in the feature extraction stage to protect sensitive data information.

3. A method for updating and querying dynamic service data according to claim 1, characterized in that: The incremental synchronization mechanism in S2 realizes asynchronous communication through a message queue middleware, and the version control technology adopts an immutable log recording method based on a hash chain to ensure the traceability and anti-tampering of data updates.

4. A method for updating and querying dynamic service data according to claim 1, characterized in that: The adaptive weight allocation algorithm in S3 dynamically adjusts the index weights according to historical query response time, data distribution density, and node load status, and preferentially selects query paths with low latency and high availability.

5. A method for updating and querying dynamic service data according to claim 1, characterized in that: The deep learning model in S5 adopts a hybrid architecture of a temporal convolutional neural network (TCN) and a long short-term memory network (LSTM), analyzes the time series features of the data stream in real time, and starts the self-repair process through a threshold trigger mechanism.

6. A system for the method of dynamically updating and querying service data according to any one of claims 1-5, characterized in that, Including: Data collection module: Real-time collect multi-source heterogeneous business data through a distributed sensor network, and perform data cleaning, encryption, and feature extraction; Dynamic update module: Based on the incremental synchronization mechanism and version control technology, realize the real-time update and consistency maintenance of business data; Query optimization module: Generate multi-dimensional indexes according to query semantic features, and optimize query paths using an adaptive algorithm; Visualization engine module: Dynamically render the query result into an interactive visualization view, and support users to customize the display granularity; Anomaly handling module: Real-time monitor system anomalies through a deep learning model, and trigger a self-repair mechanism to ensure data integrity and service continuity.

7. The system for updating and querying dynamic service data according to claim 6, characterized in that: The dynamic update module integrates a message queue middleware and a hash chain log technology, and supports asynchronous data synchronization and anti-tampering audit functions in high-concurrency scenarios.

8. A system for dynamic service data update and query according to claim 6, characterized in that: The visualization engine module is built with a dynamic rendering algorithm, supports cross-platform multi-terminal adaptation, and adjusts the spatio-temporal resolution of data display in real time according to user interaction behaviors.

Images