Real-time analysis system based on business data isolation and low-intrusive data synchronization method

Abstract

The invention discloses a real-time analysis system based on business data isolation and a low-intrusive data synchronization method, and belongs to the technical field of big data analysis and data security, and the system comprises a business data layer which is used for storing original business data and comprises a business database cluster and a business system; the synchronous agent module is deployed between the business data layer and the analysis data layer, realizes non-intrusive data capture by analyzing a database incremental log, and performs field-level encryption and fragmentation transmission on data; and the analysis data layer is used for storing the desensitized analysis data by adopting a column type storage architecture, and realizing real-time query and authority control of the encrypted data through a unified query gateway and a dynamic desensitization module. According to the method, safe, efficient and real-time data isolation analysis can be realized through innovative technical combination.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data analysis and data security, in particular to a real-time analysis system based on business data isolation and a low-intrusion data synchronization method. BACKGROUND

[0002] In today's digital era, data has become an important asset for enterprises and organizations, and data analysis is crucial for enterprise decision-making and development. However, existing data analysis solutions have many problems to be solved in practical application, as follows:

[0003] (1) Data security risks are prominent

[0004] Sensitive data leakage risk: In traditional data analysis solutions, the analysis system often directly accesses the production database, which has serious security vulnerabilities. For example, in the financial industry, customer transaction data, account information, personal identity information, and other highly sensitive data are easily leaked once the production database is illegally invaded or internal personnel violate regulations, which not only brings huge economic losses and privacy violations to customers, but also makes financial institutions face legal risks, reputation crises, and even affects the stability of the entire financial system.

[0005] Compliance requirements are difficult to meet: With the promulgation and implementation of laws and regulations such as the Data Security Law and the Personal Information Protection Law, higher requirements for data processing security and compliance have been put forward. In traditional solutions, data lacks effective physical isolation and dynamic desensitization mechanisms, making it difficult to meet the explicit requirements for sensitive data protection in regulations, resulting in compliance risks for enterprises when conducting data analysis business.

[0006] (2) Performance interference problem is serious

[0007] Business system efficiency is affected: Analysis queries usually require complex calculations and processing of large amounts of data, which will occupy a large amount of business database resources such as CPU, memory, disk I / O, etc. In busy business systems, the concurrent execution of a large number of analysis queries will significantly slow down the response speed of the business database, and even cause congestion, making the efficiency of normal business transaction processing drop significantly, seriously affecting the business operation of enterprises.

[0008] System stability is threatened: Long-term high-load analysis queries may cause business database failures such as server downtime, data loss, etc., which will further affect the stability and availability of the entire business system, causing immeasurable losses to enterprises.

[0009] (3) Data timeliness cannot meet the demand

[0010] Traditional ETL mode lags behind: The traditional ETL (Extract-Transform-Load) solution usually uses a daily batch synchronization method for data processing, which cannot reflect the latest changes in business data in real time. In today's fast-changing market environment, enterprises need to obtain data and analyze it in a timely manner to make accurate decisions. For example, in the e-commerce field, real-time analysis of user purchasing behavior and market trends is crucial for timely adjustment of marketing strategies, but the traditional daily batch synchronization method leads to a serious lag in analysis results, which cannot meet the real-time decision-making needs of enterprises.

[0011] Real-time business scenarios are limited: With the continuous emergence of real-time business scenarios such as the Internet of Things, real-time marketing, financial risk control, etc., the demand for real-time data analysis is growing. The traditional solution cannot effectively play a role in these scenarios due to poor data timeliness, limiting the expansion and innovation of enterprise business.

[0012] (4) Limitations of existing technical solutions

[0013] Patent technology has limitations: For example, patent CN202411383639.2 discloses a data desensitization method, which mainly focuses on the data desensitization technology itself, but does not solve the problem of the performance impact of real-time synchronization and analysis systems on business systems. In actual application, the real-time nature of data synchronization and the performance interference of analysis systems on business systems are two key problems that need to be solved, but this patent does not address them, resulting in certain limitations in actual application.

[0014] Invasive synchronization methods have obvious drawbacks: Traditional data synchronization technologies mostly use invasive synchronization methods based on API interfaces or modifications to business systems. This method requires modifying the code of the business system, increasing development and maintenance costs, and may introduce new system vulnerabilities, while having a significant impact on the normal operation of the business system, which cannot meet the needs of enterprises for low-invasive data synchronization.

[0015] Static desensitization scheme lacks security: Traditional static desensitization schemes desensitize data during the data storage stage. Although this method protects data privacy to some extent, sensitive data still exists in the storage layer and is at risk of persistent exposure. Once the data in the storage layer is leaked, the desensitized data may still contain valuable information, posing a threat to data security.

[0016] In summary, existing data analysis solutions have obvious shortcomings in data security, performance loss, and data timeliness, and cannot meet the needs of enterprises in high-sensitive data scenarios for "low-invasive, high-timeliness, and strong security" data analysis. A new technical solution is needed to solve these problems. SUMMARY

[0017] The technical task of the present application is to provide a real-time analysis system based on business data isolation and a low-intrusion data synchronization method to achieve safe, efficient and real-time data isolation analysis through innovative technical combination.

[0018] The technical solution adopted by the present application to solve its technical problems is:

[0019] The real-time analysis system based on business data isolation comprises:

[0020] A business data layer for storing original business data, comprising a business database cluster and a business system;

[0021] A synchronization agent module deployed between the business data layer and the analysis data layer, which realizes non-intrusive data capture by analyzing database incremental logs and performs field-level encryption and sharded transmission on the data;

[0022] An analysis data layer using columnar storage architecture to store desensitized analysis data, which realizes real-time query and permission control of encrypted data through a unified query gateway and a dynamic desensitization module.

[0023] Further, the synchronization agent module realizes incremental data capture by analyzing MySQL binlog, PostgreSQL redolog or Oracle archive log, without modifying the business system code.

[0024] Further, the business data layer and the analysis data layer are isolated by a one-way firewall, which only allows the synchronization agent module to transmit data to the analysis data layer through an SSL / TLS encrypted port (443 / 9443) and prohibits reverse communication.

[0025] The synchronization agent module has a built-in data verification mechanism that generates a hash check code (such as SHA-256) when encrypting and sharding data, and the analysis data layer performs hash comparison and timestamp verification after receiving the data to ensure that there is no loss or tampering during data transmission. For abnormal data (such as verification failure), a snapshot patch channel is automatically triggered to repair the data.

[0026] Further, the analysis data layer uses Parquet, ORC or Delta Lake columnar storage format to accelerate the scanning and aggregation of encrypted data through column-level indexing (prefix index / bitmap index).

[0027] Further, the synchronization agent module includes a dual-channel synchronization mechanism:

[0028] A real-time streaming channel that processes second-level incremental data with a delay control within 500ms;

[0029] Snapshot patching channel, through full data snapshot and hash check to realize data repair after failure;

[0030] The snapshot patching channel automatically triggers full synchronization when detecting data check failure (hash value mismatch / time stamp anomaly), and transmits only the missing data slices through differential comparison.

[0031] Further, the synchronization agent module supports AES-256, RSA-2048 or field-level dynamic encryption of the national encryption algorithm SM4, and the encryption strategy can be configured according to the data sensitivity level (top secret / confidential / ordinary).

[0032] The slice transmission mechanism splits the encrypted data into 128KB-1MB slices based on data type (structured / unstructured), timestamp (sliced by hour) or hash value (user ID modulo), and transmits them in parallel through multiple channels.

[0033] Further, the dynamic desensitization module of the analysis data layer supports a three-level permission policy:

[0034] Administrator permission: view full data (including sensitive fields);

[0035] Analysis permission: view part of the data according to the mask rule (such as the middle 8 digits of the ID number are displayed as ****);

[0036] Scientific researcher permission: only view anonymized statistical data (such as age interval / disease classification);

[0037] The dynamic desensitization engine configures field-level desensitization rules based on user roles (administrator, analyst, scientific researcher), including mask processing (such as displaying the middle 8 digits of the ID number as ****), generalization processing (such as displaying age by interval, hiding specific values), and deletion processing (directly shielding high-sensitive fields such as bank card CVV code);

[0038] The desensitization rules are real-time bound with the query statement, and are dynamically applied when the distributed computing engine (Spark / Flink) executes the query, avoiding the exposure of sensitive data in the storage layer.

[0039] Further, the unified query gateway supports the conversion of standard SQL to encrypted columnar query, integrates predicate pushdown (pushes the WHERE condition to the storage layer) and vectorized execution (batch processing of data columns) optimization.

[0040] Further, the synchronization agent module supports multi-source data access, including relational databases (MySQL / PostgreSQL / Oracle), non-relational databases (MongoDB / Cassandra) and real-time stream data (Kafka / Pulsar).

[0041] The analysis data layer realizes the elastic expansion of query tasks through a distributed computing framework (Spark / Flink), and supports dynamically adding computing nodes to cope with the growth of PB-level data volume.

[0042] The application also claims to protect a low-invasive data synchronization method based on business data isolation, which is implemented based on the above-mentioned system and includes the following steps:

[0043] (1) The synchronization agent module listens to the business database log in real time, captures the INSERT / UPDATE / DELETE incremental data change events, and performs field-level dynamic encryption (AES-256 / SM4) on the incremental data and sharding according to the data type / time stamp / hash value.

[0044] (2) The incremental data is executed field-level dynamic encryption (AES-256 / SM4), and is sharded according to the data type / time stamp / hash value.

[0045] (3) The sharded data is transmitted to the analysis data layer through a one-way encryption channel (SSL / TLS), with a SHA-256 check code.

[0046] (4) The analysis data layer checks the received data, and stores it according to the columnar storage format after decryption.

[0047] (5) The user sends an SQL request through a unified query gateway, and the dynamic desensitization module returns the result after executing field-level desensitization according to the user role.

[0048] Compared with the prior art, the real-time analysis system based on business data isolation and the low-invasive data synchronization method of the application have the following beneficial effects:

[0049] Through the deep integration of the three-layer physical isolation architecture and the low-invasive technical system, the application has achieved a breakthrough in data security, system performance, and business adaptability, and the specific improvements are as follows:

[0050] 1. Breakthrough in data security and compliance.

[0051] Build a full-life-cycle security barrier: through the three-level protection system of physical isolation architecture, field-level dynamic encryption, and real-time desensitization in the query stage, realize the "availability but invisibility" of sensitive data. In the financial transaction scenario, customer bank card numbers, transaction amounts, and other data are encrypted using AES-256 during transmission, stored in ciphertext form, and dynamically masked according to the analysis authority during query (such as displaying only the last 4 digits), thereby blocking the risk of data leakage from the source. This solution has a security improvement of more than 300% compared to traditional static desensitization solutions, and fully meets the compliance requirements of the Data Security Law and the Personal Information Protection Law for the processing of sensitive data.

[0052] Meet the special needs of high-sensitivity fields: In the medical research scene, the system supports fine-grained desensitization control of medical record data: researchers can only access desensitized diagnostic results (such as "respiratory disease" instead of specific disease names) and statistical data, and patient names, ID numbers and other information are completely shielded. This "data is available but privacy is not disclosed" mechanism supports big data analysis for medical research while protecting patient privacy and complying with industry standards.

[0053] 2. Reduce the load of business systems.

[0054] After adopting the non-invasive log analysis technology, the business database does not need to provide additional resources for analysis queries, the CPU occupancy rate is reduced from 60%-80% of the traditional scheme to below 10%, the I / O throughput is greatly improved, and the problem of resource contention between analysis queries and business transactions is completely solved.

[0055] 3. Data consistency guarantee.

[0056] The snapshot completion channel automatically triggers full data synchronization when the system is initialized or there is a network failure, accurately identifies missing data through a hash check and timestamp comparison mechanism, and completes the missing data. Ensure real-time synchronization of analysis data and business data to avoid decision-making errors caused by inconsistent data.

[0057] 4. Business adaptation and expansion advantage.

[0058] Multi-source heterogeneous data fusion capability: The synchronization agent module supports multi-source access such as relational databases (such as Oracle / MySQL), non-relational databases (such as MongoDB), real-time streaming data (such as Kafka), and realizes standardized processing through a unified data conversion engine.

[0059] Flexible permission management: The dynamic desensitization module supports field-level permission control based on user roles, for example: bank administrators can view full transaction data (including sensitive fields), risk analysts can only view desensitized transaction amounts (such as "*5678") and transaction times, and researchers can only access anonymized transaction statistical reports. This fine-grained permission control mechanism meets the differentiated data usage needs of different departments and different positions within financial institutions, while reducing the risk of data misuse.

[0060] The present application realizes the data analysis goal of "safety compliance, high efficiency and low consumption, real-time intelligence" through technical innovation, provides a solution with advanced technology and commercial feasibility for high-sensitivity data scenarios, and fills the gap in the field of physical isolation analysis and low-invasion synchronization in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1is a system architecture diagram of a real-time analysis system based on business data isolation provided by an embodiment of the present application;

[0062] Figure 2 is an incremental data crawling and encryption process diagram provided by an embodiment of the present application;

[0063] Figure 3 is a dynamic desensitization and query execution timing diagram provided by an embodiment of the present application;

[0064] Figure 4 is a schematic diagram of a dual-channel data synchronization mechanism provided by an embodiment of the present application. DETAILED DESCRIPTION

[0065] The present application will be further described below in conjunction with specific embodiments.

[0066] The embodiment of the present application provides a real-time analysis system based on business data isolation, and the system adopts a three-layer architecture design of a business data layer, a synchronization agent module and an analysis data layer, each layer is connected through a one-way encrypted data stream to form a physical isolation barrier.

[0067] The business data layer: deploy a business database cluster (such as MySQL / PostgreSQL) and a business system, store original business data, and is a data source.

[0068] The synchronization agent module: deployed between the business data layer and the analysis data layer, isolated by a one-way firewall (SSL / TLS encrypted port), and only allows data to flow from the business layer to the analysis layer. The synchronization agent module realizes non-invasive data crawling by analyzing the database incremental log, and performs field-level encryption and sharding transmission on the data.

[0069] The analysis data layer: adopts a columnar storage cluster (Parquet / ORC format) and a distributed computing framework (Spark / Flink), supports storage and real-time analysis of encrypted data. The analysis data layer stores desensitized analysis data, and realizes real-time query and permission control of encrypted data through a unified query gateway and a dynamic desensitization module.

[0070] The system realizes a one-way data flow security mechanism

[0071] Data can only flow from the business data layer to the analysis data layer through the synchronization agent module, and reverse transmission is prohibited. The business data layer and the analysis data layer are isolated by a one-way firewall, and only the synchronization agent module is allowed to transmit data to the analysis data layer through an SSL / TLS encrypted port (443 / 9443), and reverse communication is prohibited. The synchronization agent module realizes non-invasive data crawling through database log analysis, avoids direct access of the analysis system to the business database, and cuts off the data leakage risk from the architecture level.

[0072] The synchronization proxy module includes the following technologies:

[0073] (1) Non-intrusive incremental data capture:

[0074] It employs database log parsing technologies (such as MySQL binlog, PostgreSQL redo log, or Oracle) to monitor incremental changes in the business database in real time without modifying the business system code. The log parsing engine supports parallel parsing across multiple database clusters, ensuring the real-time performance and integrity of data capture.

[0075] (2) Field-level dynamic encryption and fragmented transmission:

[0076] The encryption module supports field-level dynamic encryption using AES-256, RSA-2048, or the national cryptographic algorithm SM4. Encryption strategies can be configured according to data sensitivity levels (top secret / confidential / normal), and different encryption strategies can be configured based on data sensitivity levels (such as ID card number, bank card number) to achieve field-level dynamic encryption.

[0077] The fragmented transmission manager fragments encrypted data based on data type (structured / unstructured), timestamp (fragmented by hour / day), or hash value (such as user ID hash modulo), splitting the encrypted data into fragments of 128KB-1MB. Through multi-channel parallel transmission, it improves throughput while reducing the transmission pressure of a single channel.

[0078] (3) Dual-channel synchronization mechanism:

[0079] Real-time streaming channel: processes incremental data at the second level, suitable for continuous data synchronization in normal business scenarios, with latency controlled within 500ms.

[0080] Snapshot Completion Channel: Used for full synchronization during system initialization, network failures, or data consistency checks. It automatically identifies missing data and triggers completion through data comparison and verification (hash value + timestamp), ensuring the integrity of the analyzed data. It ensures the integrity of transmitted data by comparing SHA-256 hash checksums with timestamps; automatic retransmission or snapshot completion is triggered when verification fails. The snapshot completion channel achieves data repair after failures through full data snapshots and hash verification. When data verification failure is detected (hash value mismatch / timestamp anomaly), it automatically triggers full synchronization and transmits only missing data fragments through difference comparison.

[0081] The analytical data layer includes:

[0082] (1) Columnar storage and query optimization:

[0083] Employing columnar storage formats such as Parquet, ORC, or Delta Lake allows for contiguous storage of similar fields, reducing I / O reads and improving scan efficiency. For encrypted data, column-level indexes (such as prefix indexes and bitmap indexes) accelerate the scanning and aggregation of encrypted data, thus speeding up queries.

[0084] (2) The unified query gateway supports the conversion from standard SQL to encrypted columnar queries and integrates predicate pushdown (pushing filtering conditions to the storage layer) and vectorized execution (batch processing of data columns) optimization strategies.

[0085] (3) Dynamic desensitization and access control:

[0086] The dynamic data masking module in the analysis data layer supports a three-level permission policy:

[0087] Administrator privileges: View all data (including sensitive fields);

[0088] Analyst permissions: View partial data according to masking rules (e.g., the middle 8 digits of an ID number are displayed as "****");

[0089] Researcher permissions: Only view anonymized statistical data (such as age range / disease classification).

[0090] The dynamic data masking engine configures field-level data masking rules based on user roles (administrators, analysts, researchers):

[0091] Masking strategy: For example, display the middle 8 digits of the ID number as "****";

[0092] Generalization strategy: For example, display age ranges (20-30 years old) and hide specific values;

[0093] Deletion strategy: Highly sensitive fields (such as bank card CVV codes) are directly blocked from display.

[0094] The data masking logic is bound to the query statement in real time, and data masking is performed only as needed during the query execution phase. It is dynamically applied when the distributed computing engine (Spark / Flink) executes the query, thus avoiding the persistent exposure of sensitive data in the storage layer.

[0095] The system employs the following solutions to ensure data security and integrity:

[0096] Data transmission security:

[0097] The business data layer and the analysis data layer are isolated by a one-way firewall, with only SSL / TLS encrypted ports (such as port 443) open. Data transmission is encrypted using encryption algorithms to prevent data eavesdropping and tampering.

[0098] Data integrity verification mechanism:

[0099] The synchronization proxy module generates hash checksums (such as SHA-256) during data encryption and fragmentation. After analyzing the received data from the data layer, it performs hash comparison and timestamp verification to ensure that no data is lost or tampered with during transmission. For abnormal data (such as verification failures), the snapshot completion channel is automatically triggered to repair the data.

[0100] This system enables multi-source data access.

[0101] Heterogeneous data source adaptation: The synchronization agent module supports access from multiple sources, including relational databases (MySQL / PostgreSQL / Oracle), non-relational databases (MongoDB / Cassandra), and real-time streaming data (Kafka / Pulsar). Through the unified log parsing and data transformation module, heterogeneous data is standardized into a unified format (such as JSON / Parquet), enabling multi-source data fusion analysis.

[0102] The analytical data layer uses a distributed computing framework (Spark / Flink) to enable elastic scaling of query tasks, supporting the dynamic addition of computing nodes to cope with petabyte-level data growth.

[0103] like Figure 1 The diagram shown illustrates the system architecture, demonstrating the physical isolation between the business layer, synchronization proxy, and analysis layer. The business data layer, synchronization proxy module, and analysis data layer are connected via a one-way encrypted data stream, clearly illustrating the physical isolation between each layer and the direction of data transmission.

[0104] Figure 2 The process of incremental data capture and encryption is demonstrated, including step ①: monitoring database logs and monitoring changes in the database logs of the business data layer in real time; step ②: field-level encryption, performing field-level encryption on the captured incremental data; and step ③: fragmented transmission, transmitting the encrypted data in fragments to the analysis data layer.

[0105] Figure 3 The sequence diagram for dynamic masking and query execution illustrates the workflow and time sequence of the dynamic masking module during the query process, as well as the transformation and execution process of the query statement by the unified query gateway.

[0106] Figure 4 The diagram illustrates a dual-channel data synchronization mechanism, including two synchronization methods: real-time streaming and snapshot completion. Real-time streaming handles incremental data generated in real time, while snapshot completion performs full data synchronization during system initialization or data anomalies, ensuring data integrity and consistency.

[0107] This system enables low-intrusion data synchronization, and the process is as follows:

[0108] Incremental data capture: The synchronization agent module monitors the business database logs in real time and captures incremental data records when data changes (INSERT / UPDATE / DELETE) are detected;

[0109] Field-level encryption: Based on preset sensitive field rules, the captured incremental data is encrypted using AES-256, RSA, or SM4 algorithms to ensure that sensitive information is desensitized before transmission;

[0110] Fragmentation and Verification Generation: Fragment the encrypted data according to data type, timestamp, or hash value, and generate the hash verification code and timestamp for each fragment;

[0111] One-way encrypted transmission: Fragmented data is sent to the analysis data layer via an SSL / TLS encrypted channel. The transmission process is controlled by a one-way firewall, which prohibits reverse communication.

[0112] Data verification and storage: The data analysis layer performs hash and timestamp verification on the received data. After verification, the data is decrypted and written to the storage cluster in columnar storage format (Parquet / ORC).

[0113] Real-time query and dynamic data masking: Users send SQL query requests through a unified query gateway. The system parses the SQL and generates an optimized query plan. When reading encrypted data, the dynamic data masking module is triggered. Field-level data masking is performed according to user permissions, and the results are returned.

[0114] Through the above architecture and process design, the core technical goals of "low-intrusive data synchronization, physical isolation security analysis, real-time dynamic de-identification, and efficient query optimization" have been achieved, which is suitable for compliance analysis needs in highly sensitive data scenarios such as finance, healthcare, and government.

[0115] This invention also provides a low-intrusion data synchronization method based on business data isolation. This method is implemented based on the low-intrusion data synchronization system based on business data isolation described in the above embodiments, and includes the following steps:

[0116] (1) The synchronous agent module listens to the business database logs in real time and captures INSERT / UPDATE / DELETE incremental data change events;

[0117] (2) Perform field-level dynamic encryption (AES-256 / SM4) on incremental data and shard it according to data type / timestamp / hash value;

[0118] (3) The fragmented data is transmitted to the analysis data layer via a one-way encrypted channel (SSL / TLS), with an accompanying SHA-256 checksum;

[0119] (4) The data layer analyzes the received data and, if it passes the verification, decrypts and stores it in columnar storage format;

[0120] (5) Users send SQL requests through the unified query gateway. The dynamic desensitization module performs field-level desensitization according to the user's role and then returns the results.

[0121] Through the specific embodiments described above, those skilled in the art can easily implement the present invention. However, it should be understood that the present invention is not limited to the specific embodiments described above. Based on the disclosed embodiments, those skilled in the art can arbitrarily combine different technical features to achieve different technical solutions.

[0122] Except for the technical features described in the specification, all other technologies are known to those skilled in the art.

Claims

1. A real-time analysis system based on business data isolation, characterized in that, include: The business data layer is used to store raw business data, including the business database cluster and business systems; The synchronous proxy module is deployed between the business data layer and the analysis data layer. It achieves non-intrusive data capture by parsing incremental database logs and performs field-level encryption and fragmented transmission on the data. The analysis data layer uses a columnar storage architecture to store the anonymized analysis data, and a unified query gateway and dynamic anonymization module enable real-time querying and access control of encrypted data.

2. The real-time analysis system based on business data isolation according to claim 1, characterized in that, The synchronization agent module captures incremental data by parsing MySQL binlog, PostgreSQL redo log, or Oracle archive log, without requiring modification to the business system code.

3. The real-time analysis system based on business data isolation according to claim 1, characterized in that, The business data layer and the analysis data layer are isolated by a one-way firewall, which only allows the synchronization proxy module to transmit data to the analysis data layer through the SSL / TLS encrypted port, and prohibits reverse communication. The synchronous proxy module has a built-in data verification mechanism that generates a hash check code when the data is encrypted and fragmented. After analyzing the data received by the data layer, it performs hash comparison and timestamp verification to ensure that there is no loss or tampering during data transmission. For abnormal data, the snapshot completion channel is automatically triggered to repair the data.

4. The real-time analysis system based on business data isolation according to claim 1, characterized in that, The analytical data layer uses Parquet, ORC, or Delta Lake columnar storage formats, and accelerates the scanning and aggregation of encrypted data through column-level indexes.

5. The real-time analysis system based on business data isolation according to claim 1, characterized in that, The synchronization proxy module includes a dual-channel synchronization mechanism: Real-time streaming channel, processing incremental data in seconds, with latency controlled within 500ms; The snapshot completion channel enables data repair after a failure by using full data snapshots and hash verification. When the snapshot completion channel detects a data verification failure, it automatically triggers full synchronization and transmits only the missing data fragments through difference comparison.

6. The real-time analysis system based on business data isolation according to claim 1 or 5, characterized in that, The synchronization agent module supports field-level dynamic encryption using AES-256, RSA-2048, or the national cryptographic algorithm SM4, and the encryption strategy can be configured according to the data sensitivity level. The fragmented transmission mechanism splits encrypted data into fragments ranging from 128KB to 1MB based on data type, timestamp, or hash value, and transmits them in parallel through multiple channels.

7. The real-time analysis system based on business data isolation according to claim 1, characterized in that, The dynamic data masking module in the analysis data layer supports a three-level permission policy: Administrator privileges: View all data; Analyst permissions: View partial data according to mask rules; Researchers Permissions: Only view anonymized statistics; The dynamic masking engine configures field-level masking rules based on user roles, including masking, generalization, and deletion. The de-identification rules are bound to the query statement in real time and applied dynamically when the distributed computing engine executes the query.

8. The real-time analysis system based on business data isolation according to claim 1 or 7, characterized in that, The unified query gateway supports the conversion from standard SQL to encrypted columnar queries and integrates predicate pushdown and vectorized execution optimization.

9. The real-time analysis system based on business data isolation according to claim 1, characterized in that, The synchronization agent module supports multi-source data access, including relational databases, non-relational databases, and real-time streaming data; The analytical data layer uses a distributed computing framework to enable elastic scaling of query tasks, supporting the dynamic addition of computing nodes to cope with petabyte-level data growth.

10. A low-intrusion data synchronization method based on business data isolation, characterized in that: This method, implemented based on the system described in any one of claims 1 to 9, includes the following steps: (1) The synchronous agent module listens to the business database logs in real time and captures INSERT / UPDATE / DELETE incremental data change events; (2) Perform field-level dynamic encryption on incremental data and shard it according to data type / timestamp / hash value; (3) The fragmented data is transmitted to the analysis data layer through a one-way encrypted channel, with an SHA-256 checksum attached; (4) The data layer analyzes the received data and, if it passes the verification, decrypts and stores it in columnar storage format; (5) Users send SQL requests through the unified query gateway. The dynamic desensitization module performs field-level desensitization according to the user's role and then returns the results.

Images