A columnar storage and policy engine based hierarchical data productization distribution system, method and storage medium

Abstract

The application discloses a hierarchical data productization distribution system and method based on column storage and a policy engine and a storage medium, relates to the technical field of data processing and distribution, and comprises a column storage module; the column storage module is used for partition storage and compression coding of original multi-source data according to a column storage architecture; the policy engine module is used for receiving a data grading result, automatically matching corresponding data desensitization rules, access permission rules, distribution path rules and transmission encryption rules based on a grading label, and generating an executable distribution strategy set. Through deep coupling of the column storage architecture and the data grading model, automatic mapping of the grading result to the distribution strategy is realized in combination with the policy engine, an integrated closed loop of data storage, grading, productization packaging and intelligent distribution is constructed, and the efficiency, safety and intelligent level of data productization distribution are significantly improved.

Description

Technical Field

[0001] This application relates to the field of data processing and distribution technology, and more specifically, to a hierarchical data product distribution system, method, and storage medium based on columnar storage and a strategy engine. Background Technology

[0002] With the rapid development of big data technology, enterprises have accumulated massive amounts of multi-source, heterogeneous data. How to transform raw data into reusable data products and achieve secure and efficient distribution has become a core challenge in the implementation of data assetization. Existing technologies mainly suffer from the following technical deficiencies: Firstly, traditional row-based storage architecture suffers from high IO overhead and high query latency when performing large-scale data queries and extractions for data analysis scenarios, making it unable to support the high-frequency, multi-dimensional data retrieval and processing needs in the process of data productization. Moreover, data storage and data product distribution are disconnected, and data needs to undergo multiple format conversions and data transfers from the storage layer to the distribution layer, resulting in poor timeliness and serious resource consumption.

[0003] Secondly, data classification relies heavily on manual annotation or simple rule matching, resulting in a single classification dimension. It cannot achieve dynamic classification by comprehensively considering multiple factors such as data sensitivity, business value, access frequency, and quality level. Furthermore, the classification results are disconnected from the distribution strategy, and the classification information cannot automatically drive adaptive adjustments to access control, desensitization rules, and distribution paths. The strategy configuration is cumbersome and lacks flexibility.

[0004] Third, the level of data productization is low. Most distribution systems only transmit raw data and lack standardized data product packaging mechanisms. The distribution process lacks unified scheduling and control, and cannot dynamically select the optimal distribution path and transmission method based on factors such as data level, recipient permissions, and network status. This poses a risk of data leakage and results in low distribution efficiency.

[0005] Therefore, to address the above issues, a hierarchical data productization distribution system, method, and storage medium based on columnar storage and a strategy engine are proposed. Summary of the Invention

[0006] To overcome the aforementioned deficiencies of the prior art, this application provides a hierarchical data productization and distribution system, method, and storage medium based on columnar storage and a strategy engine to solve the problems mentioned in the background art.

[0007] First aspect To achieve the above objectives, this application provides the following technical solution: a hierarchical data productization and distribution system based on columnar storage and a strategy engine, comprising a columnar storage module, a data hierarchical engine module, a strategy engine module, a data productization encapsulation module, and an intelligent distribution scheduling module: The columnar storage module is used to partition and compress the original multi-source data according to the columnar storage architecture, build the underlying data foundation for analytical workloads, and provide a columnar query interface to the outside world. The data grading engine module has a built-in multi-dimensional data grading evaluation model, which is used to extract data features from the columnar storage module, comprehensively evaluate data assets based on four dimensions: data sensitivity, business value, data quality, and access popularity, output data grade labels, and write the grading results back to the metadata column of the columnar storage. The strategy engine module has a built-in hierarchical strategy mapping rule library and dynamic rule reasoning mechanism. It is used to receive data hierarchical results, automatically match the corresponding data desensitization rules, access permission rules, distribution path rules and transmission encryption rules based on hierarchical tags, and generate an executable distribution strategy set. The data productization and encapsulation module is used to standardize the raw data according to the distribution strategy set, including data desensitization, metadata injection, product specification definition and data asset identification generation, to form standardized data products. The intelligent distribution scheduling module is used to select the optimal distribution channel and transmission mode based on the distribution strategy and receiver attributes, combined with real-time network status and link load, using an intelligent scheduling algorithm to complete the secure distribution of data products, and to monitor and audit the entire distribution link.

[0008] Furthermore, the columnar storage module includes a data partitioning unit, a compression encoding unit, a column indexing unit, and a metadata management unit; The data partitioning unit partitions the data in two layers according to the time dimension and the business dimension, and stores the data independently by column in each partition. The compression encoding unit uses differentiated compression algorithms for columns of different data types. Numeric columns use a combination of differential compression and dictionary encoding, while string columns use prefix encoding and lightweight hash compression. The column index unit creates inverted indexes and Bloom filter indexes for frequently queried columns; The metadata management unit maintains the statistical characteristics, hierarchical labels, permission identifiers, and lineage information of each column of data.

[0009] Furthermore, the data classification engine module includes a feature extraction unit, a classification evaluation unit, and a classification calibration unit; The feature extraction unit reads the data columns to be classified in batches from the columnar storage and automatically extracts features such as data field type, content pattern, frequency of occurrence, and missing rate. The graded evaluation unit performs a comprehensive score on each data table or data field based on a multi-dimensional weighted evaluation model, and maps the score range to a multi-level data grade. The grade calibration unit introduces a manual review and feedback mechanism to perform secondary calibration on the boundary score data and feed the calibration results back to the graded evaluation model for parameter iterative optimization.

[0010] Furthermore, the strategy engine module includes a rule base unit, a rule reasoning unit, and a strategy generation unit; The rule base unit uses hierarchical tags as the primary key to store predefined de-identification rules, permission rules, encryption rules, and distribution routing rules; The rule reasoning unit supports dynamic rule derivation based on condition combination. When data hits multiple hierarchical dimensions at the same time, the strictest principle is used to merge rules and resolve conflicts. The strategy generation unit transforms the reasoned rules into executable strategy instructions and sends them to the data productization and encapsulation module and the intelligent distribution and scheduling module.

[0011] Furthermore, the data productization and encapsulation module includes a de-identification execution unit, a product definition unit, and an asset identification unit; The desensitization execution unit executes corresponding desensitization algorithms on data fields of different levels according to the policy instructions. High-level data is desensitized irreversibly, medium-level data is masked reversibly, and low-level data retains its original value. The product definition unit is a unified metadata description configured for data products, including product name, data range, update frequency, and usage instructions; The asset identification unit generates a globally unique digital asset identifier for each data product, and associates it with the corresponding hierarchical information and authorization information.

[0012] Furthermore, the intelligent distribution and scheduling module includes a channel management unit, a scheduling decision unit, a transmission execution unit, and an audit log unit; The channel management unit maintains multiple heterogeneous distribution channels, including direct push, API gateway, message queue and offline media transmission; The scheduling decision unit selects the optimal distribution channel and transmission parameters based on data level, receiver permissions, data volume, and network status using a multi-attribute decision algorithm. The transmission execution unit executes data distribution according to the selected channel, and the corresponding level of encryption algorithm is used throughout the transmission process; The audit log unit records the entire distribution chain operation, forming an immutable distribution audit chain.

[0013] Second aspect A hierarchical data product distribution method based on columnar storage and a strategy engine, applied to a hierarchical data product distribution system based on columnar storage and a strategy engine, includes the following steps: S1. Data columnar storage construction: Collect raw data from multiple sources, write data according to columnar storage architecture, perform partitioning, compression and index construction, and establish a metadata system; S2, Multi-dimensional intelligent data classification: Extract data features from columnar storage, calculate a comprehensive score through a multi-dimensional classification evaluation model, map and generate data level labels, and write back metadata; S3. Hierarchical-driven automatic policy generation: The policy engine receives hierarchical tags, matches them with the rule base, and performs dynamic rule reasoning to generate a complete set of distribution policies, including de-identification, permissions, encryption, and routing. S4. Standardized Packaging of Data Products: Based on the distribution strategy, the original data is desensitized, standardized metadata is injected, asset identifiers are bound, and the data is packaged into standardized data products. S5, Intelligent Scheduling and Secure Distribution: Receives distribution requests, combines data level, recipient attributes and real-time network status, schedules the optimal distribution channel to complete the data product distribution, and records full-link audit logs.

[0014] Furthermore, the execution process of the multi-dimensional hierarchical evaluation model described in S2 is as follows: First, we construct a hierarchical indicator system, which includes four primary dimensions: data sensitivity, business value, data quality, and access popularity. Each primary dimension has several secondary indicators. Then, the secondary indicators are normalized, and the weight coefficients of each dimension are determined by the analytic hierarchy process. Next, each data asset is scored item by item and the scores are weighted and summed to obtain a comprehensive score. Finally, based on the preset scoring range thresholds, the data assets are divided into multiple levels, and the number of levels is configurable.

[0015] Furthermore, the dynamic rule reasoning process described in S3 is as follows: When a single data asset corresponds to multiple hierarchical tags, the strategy engine reads the rule set corresponding to each tag; Rule conflict detection and merging are performed according to rule priority and the strictest security principles; Perform integrity checks on the merged rules to ensure that the four categories of rules—de-identification, permissions, encryption, and routing—are complete. The final output is a structured set of policy instructions for downstream modules to parse and execute. The intelligent scheduling described in S5 adopts a multi-attribute decision-making mechanism. The decision attributes include data security level, data product volume, receiver network bandwidth, current channel load, and channel security level. The comprehensive score of each candidate channel is obtained by normalizing and weighting each attribute, and the channel with the highest score is selected as the distribution path.

[0016] Third aspect A storage medium storing a computer program, which is read and executed by a processor and applied to a hierarchical data product distribution system based on columnar storage and a strategy engine.

[0017] The technical effects and advantages of this application are as follows: 1. Compared with existing technologies, this hierarchical data product distribution system, method and storage medium based on columnar storage and strategy engine adopts an integrated storage and distribution architecture, which greatly improves processing efficiency: This invention uses columnar storage as the data foundation, and hierarchical computing, product packaging and distribution scheduling are all directly based on columnar storage, avoiding repeated data transfer and format conversion between different systems; at the same time, the high compression ratio and column-level query capability of columnar storage significantly reduce the IO overhead of data reading and processing, and the overall data product distribution efficiency is significantly improved.

[0018] 2. Multi-dimensional intelligent grading model for more accurate and reasonable grading results: Breaking through the limitations of traditional single-rule grading, a quantitative evaluation model is constructed by comprehensively considering four dimensions: sensitivity, value, quality, and popularity. A manual calibration feedback mechanism is introduced to achieve model self-optimization, making the grading results more in line with actual business needs. The grading results are directly written back to the columnar storage metadata column to achieve permanent binding of grading information with data assets.

[0019] 3. Deep integration of hierarchical and policy levels to achieve automatic policy adaptation: The policy engine establishes an automatic mapping relationship between hierarchical tags and distribution policies, supports dynamic reasoning and conflict resolution of multiple rules, and automatically generates corresponding desensitization, permission, encryption and routing policies once the data level is determined, without the need for manual configuration one by one, greatly improving the efficiency and consistency of policy configuration.

[0020] 4. End-to-end productization and security management, ensuring secure and controllable distribution process: Construct a standardized data product encapsulation mechanism to transform raw data into standardized data products with metadata, asset identification, and security attributes; Combined with intelligent distribution scheduling algorithms, dynamically select distribution channels and encryption strength according to data level, ensuring full traceability and auditability, and achieving secure management and control of the entire data distribution lifecycle. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the method flow of this application; Figure 2 This is a schematic diagram of the system flow of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] Example 1 Please see Figure 1 To achieve the above objectives, this application provides the following technical solution: a hierarchical data productization and distribution system based on columnar storage and a strategy engine, comprising a columnar storage module, a data hierarchical engine module, a strategy engine module, a data productization encapsulation module, and an intelligent distribution scheduling module: The columnar storage module is used to partition and compress raw multi-source data according to the columnar storage architecture, build the underlying data foundation for analytical workloads, and provide columnar query interfaces to the outside world. The columnar storage module includes data partitioning units, compression encoding units, column indexing units, and metadata management units; The data partitioning unit performs two-level partitioning of data according to the time dimension and the business dimension. Each partition is stored independently by column. The first level is logical partitioning according to the business domain, and the second level is physical partitioning according to the time granularity. Each data column in each physical partition is stored and managed independently. The compression encoding unit uses differentiated compression algorithms for columns of different data types. Numeric columns use a combination of differential compression and dictionary encoding, while string columns use prefix encoding and lightweight hash compression. For integer and floating-point numerical sequences, difference compression combined with bit-packing encoding is preferred. The compression calculation process is as follows: Let the ordered sequence of a numerical sequence be... ,in, To calculate the total number of rows in this column, first perform a difference calculation on the sequence to obtain the difference sequence: The difference sequence is encoded using bit packing, and the maximum absolute value of the difference sequence is calculated. Calculate the required storage width Each difference is calculated using... Bit-width binary storage replaces the original fixed-width storage.

[0024] The formula for calculating the compression ratio of this numerical series is: ,in For the fixed storage width of the original data, The smaller the value, the higher the compression ratio.

[0025] For string columns, dictionary encoding combined with prefix compression is used to construct a string dictionary table Map, which maps each unique string to an integer encoding. At the same time, common prefixes are extracted from the strings in the dictionary table to further reduce storage space. The total storage size after compression is the sum of the dictionary table size and the encoding column size. The column index unit creates inverted indexes and Bloom filter indexes for frequently queried columns. Specifically, Bloom filter indexes are created for equality query columns to quickly filter data blocks that do not meet the conditions, and partitioned statistical indexes are created for range query columns. The false positive rate of the Bloom filter meets the following requirements: in, The number of hash functions. The number of elements to be inserted. The length of the bit array can be configured to keep the false positive rate below one in a thousand, significantly reducing the I / O for reading invalid data blocks; The metadata management unit maintains the statistical characteristics, hierarchical labels, permission identifiers, and lineage information of each column of data. The metadata itself is also stored in a columnar format, supporting fast retrieval.

[0026] The data grading engine module has a built-in multi-dimensional data grading evaluation model, which is used to extract data features from the columnar storage module, and quantitatively evaluate data assets based on four dimensions: data sensitivity, business value, data quality, and access popularity. It outputs data grade labels and writes the grading results back to the metadata column of the columnar storage. The data grading engine module includes a feature extraction unit, a grading evaluation unit, and a grading calibration unit; The feature extraction unit reads the data columns to be classified in batches from the columnar storage, automatically extracts the data field type, content mode, frequency of occurrence, and missing rate features, and connects with the business system to obtain information on the business line to which the data belongs, access frequency, and usage scenario. The tiered evaluation unit uses a multi-dimensional weighted evaluation model to comprehensively score each data table or data field. Based on the score range, it maps to multiple data levels. Using the data table or data field as the evaluation granularity, it comprehensively quantifies four dimensions: Data Sensitivity (scored based on whether the field content contains personal information, trade secrets, or core operational data); Business Value (scored based on the importance of the data to the business, frequency of use, and degree of decision support); Data Quality (scored based on data completeness, accuracy, consistency, and timeliness); and Access Popularity (scored based on the number of queries, downloads, and citations in a recent period). The four dimensions are weighted and summed to obtain a comprehensive score, which is then mapped to four data levels (Level 1 to Level 4) according to preset thresholds, with Level 1 being the highest level. The grading calibration unit introduces a manual review and feedback mechanism to perform secondary calibration on the boundary scoring data and feed the calibration results back to the grading assessment model for parameter iterative optimization. After grading is completed, the grading label is written back to the corresponding data column of the metadata management unit and persistently bound to the data assets.

[0027] The strategy engine module has a built-in hierarchical strategy mapping rule library and dynamic rule reasoning mechanism. It is used to receive data hierarchical results, automatically match the corresponding data desensitization rules, access permission rules, distribution path rules and transmission encryption rules based on hierarchical tags, and generate an executable distribution strategy set. The strategy engine module includes a rule base unit, a rule reasoning unit, and a strategy generation unit; The rule base unit uses hierarchical tags as the primary key to store predefined de-identification rules, permission rules, encryption rules, and distribution routing rules. Specifically, a rule mapping table is constructed using data level as the primary key. Each data level corresponds to a complete set of distribution rules, including: de-identification rules, which define the de-identification algorithms and de-identification strengths for different fields; permission rules, which define the roles and operation permissions that can access the data; encryption rules, which define the encryption algorithm levels for transmission and storage; and routing rules, which define the allowed distribution channels and transmission restrictions. The rule reasoning unit supports dynamic rule derivation based on condition combinations. When data simultaneously hits multiple hierarchical dimensions, rules are merged and conflicts are resolved according to the strictest principle. Specifically, when the same data product contains multiple data fields of different levels, or when the same data simultaneously hits multiple business rules, rules are merged according to the strictest security principle, i.e., the highest desensitization strength, the smallest permission range, the highest encryption level, and the most secure distribution channel are selected. At the same time, rule conflict detection is performed. When rules of different dimensions contradict each other, they are resolved based on a preset priority sequence. The strategy generation unit transforms the reasoned rules into executable strategy instructions, which are then distributed to the data productization and encapsulation module and the intelligent distribution and scheduling module. The instructions explicitly specify the execution object, execution method, and constraints for each operation.

[0028] The data productization and packaging module is used to standardize the raw data according to the distribution strategy set, including data desensitization, metadata injection, product specification definition and data asset identification generation, to form standardized data products; The data productization and encapsulation module includes a de-identification execution unit, a product definition unit, and an asset identification unit; The data masking execution unit executes corresponding data masking algorithms on different levels of data fields according to policy instructions. High-level data uses irreversible masking, medium-level data uses reversible masking, and low-level data retains the original value. Specifically, it parses the masking rules in the policy instructions and performs differentiated masking processing on fields of different levels: Level 1 data fields use irreversible hash masking or generalization processing to ensure that the original value cannot be restored; Level 2 data fields use reversible masking or partial masking processing, which authorized users can restore through a key; Level 3 data fields only undergo format standardization and retain the original business value; Level 4 data is public data and does not require masking. The product definition unit configures a unified metadata description for data products, including product name, data scope, update frequency, and usage instructions. Specifically, it also includes a unique product code, product name, data content description, data coverage, update cycle, data format, and usage instructions, and automatically associates the corresponding data level and security instructions. The asset identification unit generates a globally unique digital asset identifier for each data product, which is associated with the corresponding hierarchical information and authorization information. Specifically, a distributed identifier generation algorithm is used to generate a globally unique digital asset ID for each data product. This ID is used throughout the entire distribution process and serves as the sole basis for permission verification, audit traceability, and billing statistics. The intelligent distribution scheduling module is used to select the optimal distribution channel and transmission mode based on the distribution strategy and recipient attributes, combined with real-time network status and link load, using an intelligent scheduling algorithm to complete the secure distribution of data products, and to monitor and audit the entire distribution link.

[0029] The intelligent distribution and scheduling module includes a channel management unit, a scheduling decision unit, a transmission execution unit, and an audit log unit; The channel management unit maintains multiple heterogeneous distribution channels, including direct push, API gateway, message queue and offline media transmission. Each channel is marked with a security level, transmission rate limit and current load status. The scheduling decision unit selects the optimal distribution channel and transmission parameters based on data level, receiver permissions, data volume, and network status using a multi-attribute decision algorithm. Specifically, when a distribution request is received, the unit reads the security level, data volume, receiver permissions, and network status of the data product, and comprehensively scores the security matching degree, transmission efficiency, and load of each candidate channel, selecting the channel with the highest comprehensive score as the distribution path. For ultra-large data products, breakpoint resume and fragmented transmission mechanisms are automatically enabled. The transmission execution unit sends data according to the selected channel. The corresponding level of encryption algorithm is used throughout the transmission process, and the receiver decrypts and reads the data using the authorized key. The audit log unit records the entire distribution operation, forming an immutable distribution audit chain. Specifically, it logs all nodes in the chain, including distribution requests, policy matching, channel selection, transmission start, transmission end, and reception confirmation. The logs adopt a chain storage structure to ensure immutability and support post-event auditing and source tracing analysis.

[0030] Example 2 Please see Figure 2 A hierarchical data product distribution method based on columnar storage and a strategy engine, applied to a hierarchical data product distribution system based on columnar storage and a strategy engine, includes the following steps: S1. Data columnar storage construction: Collect raw data from multiple sources, write the data according to the columnar storage architecture, perform partitioning, compression and index construction, and establish a metadata system. The raw data from multiple sources comes from business systems, log systems and external data sources. After data cleaning and format standardization, it is written to the columnar storage engine. The detailed strategy is as follows: During the write process, the data partitioning unit performs partition routing according to both business domain and time dimensions, writing the data to the corresponding physical partition; the compression encoding unit determines the data type column by column and automatically selects the optimal compression algorithm for encoding and compression; the column indexing unit automatically builds corresponding indexes for preset high-frequency query columns; and the metadata management unit synchronously collects statistical and attribute information for each column to generate initial metadata records. After construction is complete, the data is physically stored column by column, supporting fast reading and calculation by column. S2, Multi-dimensional intelligent data classification: Extract data features from columnar storage, calculate a comprehensive score through a multi-dimensional classification evaluation model, map and generate data level labels, and write back metadata; The execution process of the multi-dimensional hierarchical evaluation model in S2 is as follows: First, we construct a hierarchical indicator system, which includes four primary dimensions: data sensitivity, business value, data quality, and access popularity. Each primary dimension has several secondary indicators. The specific strategy for the tiered indicator system is as follows: The data sensitivity dimension includes three secondary indicators: field content sensitivity level, data aggregation sensitivity, and number of data subjects; the business value dimension includes three secondary indicators: business line importance, decision support strength, and richness of use cases; the data quality dimension includes four secondary indicators: data integrity, data accuracy, data consistency, and data timeliness; and the access popularity dimension includes two secondary indicators: query access frequency and download / citation count.

[0031] Let the set of data assets to be evaluated be... ,common Item data assets; the first Data assets The The original value of the secondary indicator is ,in , This represents the total number of secondary indicators; in this example, J=12. Then, the secondary indicators are normalized, and the weight coefficients of each dimension are determined by the analytic hierarchy process. Since the dimensions, value ranges, and optimization directions of each secondary indicator differ, normalization is required first to map all indicator values ​​to the [0,1] interval. Based on their attributes, indicators are categorized into two types: positive indicators and negative indicators. Positive metrics: The higher the metric value, the better the performance of the corresponding dimension, such as business line importance, data integrity, and query access frequency; Inverse metrics: The smaller the metric value, the better the corresponding dimension performs, such as data missing rate, update delay time, etc.

[0032] The normalization formula for positive indicators is: ; The normalization formula for the contrarian indicator is: ; in, For the i-th data asset The normalized value of the secondary indicator, The first among all data assets The maximum value of the secondary indicator. For the first The minimum value of the secondary indicator. When At that time, take To avoid division by zero exceptions; The weight coefficients of each level of indicators are determined using the Analytic Hierarchy Process (AHP). The specific process is as follows: Constructing the criterion-level judgment matrix: Invite domain experts to conduct pairwise comparisons of the relative importance of the four dimensions, constructing a 4th-order judgment matrix A, with matrix elements... Indicates the first The dimension relative to the first The importance scale of each dimension; Calculate the weight vector: Solve for the maximum eigenvalue of the judgment matrix A to obtain the corresponding normalized eigenvector, which is the criterion layer weight vector. ;in, For the weight of the data sensitivity dimension, As a weight for the business value dimension, Weights for data quality dimensions As the weight of the access popularity dimension, satisfying And all weight values ​​are within the range of (0,1).

[0033] Consistency check: Calculate the consistency index CI and the consistency ratio CR. ;in To determine the largest eigenvalue of a matrix, Let be the order of the matrix. This is the average random consistency index. When... If the consistency of the judgment matrix is ​​acceptable, the weights are valid; otherwise, the judgment matrix needs to be adjusted until the consistency requirement is met.

[0034] Using the same method, the weight vectors of each secondary indicator under each dimension are calculated. Taking the data sensitivity dimension as an example, the weight vectors of its three secondary indicators are as follows: ,satisfy Similarly, the weight vector for the business value dimension can be obtained. Data quality dimension weight vector Access popularity dimension weight vector ; Next, each data asset is scored item by item and the scores are weighted and summed to obtain a comprehensive score. Specifically, the process involves: first, calculating the individual scores for each dimension; taking the data sensitivity dimension as an example, the first... Sensitivity score of data assets This is the weighted sum of the normalized values ​​of all secondary indicators under this dimension: ;in In this embodiment, the number of secondary indicators for the data sensitivity dimension is... .

[0035] Similarly, the business value dimension score can be obtained. Data quality dimension score Access popularity score .

[0036] Based on this, calculate the first... Comprehensive score of data assets ; ; The comprehensive score ranges from [0,1]. A higher score indicates a higher overall level of data and higher security control requirements. Finally, based on the preset scoring range threshold, the data assets are divided into multiple levels, and the number of levels is configurable. There are four preset data levels, corresponding to three scoring thresholds. ,satisfy The level mapping rules are as follows: when At that time, it corresponds to Level 1 data (the highest level); when At that time, it corresponds to the second-level data; when At that time, it corresponds to three levels of data; when At that time, it corresponds to Level 4 data (public level).

[0037] threshold It can be flexibly configured according to the enterprise's data security policy.

[0038] For those whose overall score is at the grade threshold boundary Data assets within the specified scope are automatically entered into the manual calibration queue. After receiving the manual review results, the grading calibration unit uses the gradient descent algorithm to iteratively optimize the model weights. The loss function is defined as the mean square error between the model's predicted grade and the manually labeled grade. ;in To manually calibrate the sample quantity, The corresponding level quantization values ​​are manually labeled, and the weights are updated along the negative gradient direction of the loss function. The iteration continues until the loss function converges, thereby achieving continuous self-optimization of the hierarchical model.

[0039] The resulting hierarchical tags are written back to the metadata columns of the columnar storage and persistently bound to the data assets.

[0040] S3. Hierarchical-driven automatic policy generation: The policy engine receives hierarchical tags, matches them with the rule base, and performs dynamic rule reasoning to generate a complete set of distribution policies, including de-identification, permissions, encryption, and routing. Each rule stored in the rule base is formally defined using triples: ;in: This is a set of rule conditions, including constraints such as data hierarchical labels, business domains, and receiver types. Define the specific execution strategy corresponding to the rule for the set of rule actions; This represents the rule priority, and its value is a positive integer. The larger the value, the higher the rule priority. The dynamic rule reasoning process in S3 is as follows: When a single data asset corresponds to multiple hierarchical tags, the strategy engine reads the rule set corresponding to each tag; For the data asset currently pending processing, its attribute set is denoted as: If and only if satisfy When all constraints are in the rule, Activated and added to the candidate rule set. : ;in This represents the total number of rules in the rule base. This indicates that the set of attributes satisfies the rule conditions and constraints; Rule conflict detection and merging are performed according to rule priority and the strictest security principles; Perform integrity checks on the merged rules to ensure that the four categories of rules—de-identification, permissions, encryption, and routing—are complete. The candidate rule set is divided into four mutually exclusive subclasses based on the rule action type: Desensitization rule set Includes all data anonymization rules; Permission rule set : Contains all access control related rules; Encryption rule set Includes all rules related to encryption of transmission and storage; Routing rule set : Contains all rules related to the selection of distribution channels.

[0041] Conflict detection is performed within each subclass. If two rules apply to the same object and their actions are mutually exclusive, they are considered to be in conflict. Taking de-identification rules as an example, if two rules specify different de-identification algorithms for the same field, then a conflict exists.

[0042] If a conflict occurs, a two-level conflict resolution mechanism is adopted, as follows: First, a security strength assessment function is defined for each type of rule to quantify the security control strength of the rule action: Desensitization rule security strength function The intensity of the desensitization is quantified based on the irreversibility of the desensitization algorithm and the degree of information loss; the intensity increases with the level of desensitization. Security strength function for permission rules The scope of accessible roles and operation permissions is quantified; the smaller the permission range, the greater the strength. Encryption rule security strength function The encryption level is quantified based on the encryption algorithm strength and key length; the higher the encryption level, the greater the strength. Routing rule security strength function Based on the channel security level and the degree of isolation of the transmission environment, the higher the security of the channel, the greater the strength.

[0043] For conflicting rules within the same subclass, the rule with the highest security strength is selected first; if the security strengths are the same, the rule with the highest priority is chosen. Higher-level rules. Taking the desensitization rule set as an example, the final selected desensitization rules... satisfy: ;in This is the priority weighting coefficient, used to balance the impact of security strength and rule priority.

[0044] Using the same method, the optimal permission rules were obtained respectively. Optimal encryption rules Optimal routing rules .

[0045] The final output is a structured set of policy instructions for downstream modules to parse and execute. To ensure logical consistency between different types of rules, a rule dependency matrix is ​​established for consistency verification. A set of rule dependency constraints, Const, is defined; for example, "high-level encryption must correspond to a high-security-level distribution channel" is a dependency constraint. The verification formula is as follows: ;in, ; The final set of rules after resolution. Represents the logical AND operation. When If true, the rule set passes the consistency check; if it fails, the rules are adjusted in descending order of priority until the check passes.

[0046] The validated rule set is ultimately transformed into a structured policy instruction set, which is then distributed to downstream modules in a standardized format. This policy instruction set ensures completeness, consistency, and executability. Furthermore, the rule base supports visual configuration and hot updates, allowing administrators to dynamically add, modify, or deactivate rules based on business needs, with rule changes taking effect immediately.

[0047] S4. Standardized Packaging of Data Products: Based on the distribution strategy, the original data is desensitized, standardized metadata is injected, asset identifiers are bound, and the data is packaged into standardized data products. The specific strategy for standardized packaging of data products is as follows: the data product packaging module receives the distribution strategy instructions, reads the target data from the columnar storage, and performs data processing according to the strategy requirements.

[0048] First, perform data masking, applying the corresponding data masking algorithms to fields of different levels. Then, data format standardization and quality verification are performed to ensure that the output data conforms to product specifications. Next, standardized product metadata is injected, including product description, version information, classification identifiers, and usage instructions; Finally, a globally unique digital asset identifier is assigned to the data product, and the data content, metadata, and asset identifier are packaged into a complete data product entity and stored in the product library for distribution. S5, Intelligent Scheduling and Secure Distribution: Receives distribution requests, combines data level, recipient attributes and real-time network status, schedules the optimal distribution channel to complete the data product distribution, and records full-link audit logs.

[0049] In S5, intelligent scheduling adopts a multi-attribute decision-making mechanism. The decision attributes include data security level, data product volume, receiver network bandwidth, current channel load, and channel security level. By normalizing and weighting each attribute, a comprehensive score for each candidate channel is obtained, and the channel with the highest score is selected as the distribution path. The multi-attribute decision-making mechanism comprehensively evaluates candidate distribution channels based on multiple decision attributes to select the optimal distribution path. The specific strategy is as follows: Candidate Channels and Decision Attributes Definition Let the set of all distribution channels maintained by the system be . ,common One channel. Based on the security level of the data product and the recipient's permissions, channels that do not meet the basic security requirements are first filtered out, resulting in a set of candidate channels. .

[0050] The decision attribute set contains 6 core attributes, and the definition and optimization direction of each attribute are as follows: Data security matching degree The degree of matching between channel security level and data security level, a positive indicator; Receiver effective bandwidth : Available network bandwidth for the receiver to access this channel, in Mbps, a positive indicator; Current channel load rate The ratio of the current transmission volume to the maximum capacity of the channel; a reverse indicator. Channel security level The security protection level of the passage itself, a positive indicator; Unit transmission cost Transmission cost per unit of data, a reverse indicator; Data volume fit The channel's ability to adapt to the current volume of data products is a positive indicator.

[0051] Let the first Candidate channels The The original value of the item attribute is ,in , The number of candidate channels. .

[0052] 2. Attribute normalization processing All attribute values ​​are normalized and uniformly mapped to the [0,1] interval: positive attributes Normalization formula: ; Reverse properties Normalization formula: ;in For the first Channel No. The normalized value of the item attribute, For all candidate channels The maximum value of the item attribute, It is the minimum value.

[0053] 3. Calculation of Overall Score Preset decision attribute weight vector: satisfy The weights can be flexibly configured according to the enterprise's business strategy and security requirements.

[0054] No. The overall fit score of the candidate channels The calculation formula is: ; 4. Optimal Channel Selection Select the channel with the highest overall score as the optimal distribution channel: ; After determining the optimal channel, the optimal number of data fragments and concurrent transmissions are automatically calculated based on the data product volume and the effective bandwidth of the channel. ;in Total size of data products This represents the maximum transmission bandwidth of the channel. For single-fragment target transmission duration, The number of data fragments is determined, and distribution efficiency is improved by transmitting fragments in parallel.

[0055] 5. Dynamic channel migration mechanism The channel status is monitored in real time during the distribution process, and the channel migration process is automatically initiated when any of the following trigger conditions are met: Channel load rate ,in This is the load threshold; Actual transmission rate ,in The minimum rate threshold; The channel is malfunctioning or a security alarm has been triggered.

[0056] After the migration is triggered, the comprehensive score of the remaining candidate channels is recalculated, the second-best channel is selected to execute the unfinished transmission task, and the breakpoint resume mechanism is enabled to ensure the reliability and continuity of the distribution task.

[0057] All operational nodes throughout the distribution process are written to the audit log, forming a complete distribution traceability chain. After distribution is completed, the recipient decrypts the data product using the authorized key and uses it.

[0058] Example 3 A storage medium storing a computer program, which is read and executed by a processor, and applied to a hierarchical data product distribution system based on columnar storage and a strategy engine.

[0059] Finally: The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A hierarchical data product distribution system based on columnar storage and a strategy engine, characterized in that: It includes a columnar storage module, a data hierarchical engine module, a strategy engine module, a data productization and encapsulation module, and an intelligent distribution and scheduling module: The columnar storage module is used to partition and compress the original multi-source data according to the columnar storage architecture, build the underlying data foundation for analytical workloads, and provide a columnar query interface to the outside world. The data grading engine module has a built-in multi-dimensional data grading evaluation model, which is used to extract data features from the columnar storage module, comprehensively evaluate data assets based on four dimensions: data sensitivity, business value, data quality, and access popularity, output data grade labels, and write the grading results back to the metadata column of the columnar storage. The strategy engine module has a built-in hierarchical strategy mapping rule library and dynamic rule reasoning mechanism. It is used to receive data hierarchical results, automatically match the corresponding data desensitization rules, access permission rules, distribution path rules and transmission encryption rules based on hierarchical tags, and generate an executable distribution strategy set. The data productization and encapsulation module is used to standardize the raw data according to the distribution strategy set, including data desensitization, metadata injection, product specification definition and data asset identification generation, to form standardized data products. The intelligent distribution scheduling module is used to select the optimal distribution channel and transmission mode based on the distribution strategy and receiver attributes, combined with real-time network status and link load, using an intelligent scheduling algorithm to complete the secure distribution of data products, and to monitor and audit the entire distribution link.

2. The hierarchical data product distribution system based on columnar storage and a strategy engine according to claim 1, characterized in that: The columnar storage module includes a data partitioning unit, a compression encoding unit, a column indexing unit, and a metadata management unit. The data partitioning unit partitions the data in two layers according to the time dimension and the business dimension, and stores the data independently by column in each partition. The compression encoding unit uses differentiated compression algorithms for columns of different data types. Numeric columns use a combination of differential compression and dictionary encoding, while string columns use prefix encoding and lightweight hash compression. The column index unit creates inverted indexes and Bloom filter indexes for frequently queried columns; The metadata management unit maintains the statistical characteristics, hierarchical labels, permission identifiers, and lineage information of each column of data.

3. The hierarchical data product distribution system based on columnar storage and a strategy engine according to claim 1, characterized in that: The data classification engine module includes a feature extraction unit, a classification evaluation unit, and a classification calibration unit. The feature extraction unit reads the data columns to be classified in batches from the columnar storage and automatically extracts features such as data field type, content pattern, frequency of occurrence, and missing rate. The graded evaluation unit performs a comprehensive score on each data table or data field based on a multi-dimensional weighted evaluation model, and maps the score range to a multi-level data grade. The grade calibration unit introduces a manual review and feedback mechanism to perform secondary calibration on the boundary score data and feed the calibration results back to the graded evaluation model for parameter iterative optimization.

4. A hierarchical data product distribution system based on columnar storage and a strategy engine as described in claim 3, characterized in that: The strategy engine module includes a rule base unit, a rule reasoning unit, and a strategy generation unit; The rule base unit uses hierarchical tags as the primary key to store predefined de-identification rules, permission rules, encryption rules, and distribution routing rules; The rule reasoning unit supports dynamic rule derivation based on condition combination. When data hits multiple hierarchical dimensions at the same time, the strictest principle is used to merge rules and resolve conflicts. The strategy generation unit transforms the reasoned rules into executable strategy instructions and sends them to the data productization and encapsulation module and the intelligent distribution and scheduling module.

5. A hierarchical data product distribution system based on columnar storage and a strategy engine as described in claim 4, characterized in that: The data productization and packaging module includes a de-identification execution unit, a product definition unit, and an asset identification unit; The desensitization execution unit executes corresponding desensitization algorithms on data fields of different levels according to the policy instructions. High-level data is desensitized irreversibly, medium-level data is masked reversibly, and low-level data retains its original value. The product definition unit is a unified metadata description configured for data products, including product name, data range, update frequency, and usage instructions; The asset identification unit generates a globally unique digital asset identifier for each data product, and associates it with the corresponding hierarchical information and authorization information.

6. A hierarchical data product distribution system based on columnar storage and a strategy engine as described in claim 5, characterized in that: The intelligent distribution and scheduling module includes a channel management unit, a scheduling decision unit, a transmission execution unit, and an audit log unit; The channel management unit maintains multiple heterogeneous distribution channels, including direct push, API gateway, message queue and offline media transmission; The scheduling decision unit selects the optimal distribution channel and transmission parameters based on data level, receiver permissions, data volume, and network status using a multi-attribute decision algorithm. The transmission execution unit executes data distribution according to the selected channel, and the corresponding level of encryption algorithm is used throughout the transmission process; The audit log unit records the entire distribution chain operation, forming an immutable distribution audit chain.

7. A hierarchical data product distribution method based on columnar storage and a strategy engine, applied to a hierarchical data product distribution system based on columnar storage and a strategy engine as described in any one of claims 1-6, characterized in that, Includes the following steps: S1. Data columnar storage construction: Collect raw data from multiple sources, write data according to columnar storage architecture, perform partitioning, compression and index construction, and establish a metadata system; S2, Multi-dimensional intelligent data classification: Extract data features from columnar storage, calculate a comprehensive score through a multi-dimensional classification evaluation model, map and generate data level labels, and write back metadata; S3. Hierarchical-driven automatic policy generation: The policy engine receives hierarchical tags, matches them with the rule base, and performs dynamic rule reasoning to generate a complete set of distribution policies, including de-identification, permissions, encryption, and routing. S4. Standardized Packaging of Data Products: Based on the distribution strategy, the original data is desensitized, standardized metadata is injected, asset identifiers are bound, and the data is packaged into standardized data products. S5, Intelligent Scheduling and Secure Distribution: Receives distribution requests, combines data level, recipient attributes and real-time network status, schedules the optimal distribution channel to complete the data product distribution, and records full-link audit logs.

8. The hierarchical data product distribution method based on columnar storage and a strategy engine according to claim 7, characterized in that: The execution process of the multi-dimensional hierarchical evaluation model described in S2 is as follows: First, we construct a hierarchical indicator system, which includes four primary dimensions: data sensitivity, business value, data quality, and access popularity. Each primary dimension has several secondary indicators. Then, the secondary indicators are normalized, and the weight coefficients of each dimension are determined by the analytic hierarchy process. Next, each data asset is scored item by item and the scores are weighted and summed to obtain a comprehensive score. Finally, based on the preset scoring range thresholds, the data assets are divided into multiple levels, and the number of levels is configurable.

9. A hierarchical data product distribution method based on columnar storage and a strategy engine according to claim 7, characterized in that: The dynamic rule reasoning process described in S3 is as follows: When a single data asset corresponds to multiple hierarchical tags, the strategy engine reads the rule set corresponding to each tag; Rule conflict detection and merging are performed according to rule priority and the strictest security principles; Perform integrity checks on the merged rules to ensure that the four categories of rules—de-identification, permissions, encryption, and routing—are complete. The final output is a structured set of policy instructions for downstream modules to parse and execute. The intelligent scheduling described in S5 adopts a multi-attribute decision-making mechanism. The decision attributes include data security level, data product volume, receiver network bandwidth, current channel load, and channel security level. The comprehensive score of each candidate channel is obtained by normalizing and weighting each attribute, and the channel with the highest score is selected as the distribution path.

10. A storage medium, characterized in that, The storage medium stores a computer program, which is read and executed by a processor and applied to the hierarchical data product distribution system based on columnar storage and a strategy engine as described in any one of claims 1-6.

Images