A data staging system and method

Abstract

The application provides a data hierarchical processing system and method, comprising a data acquisition module, a hierarchical judgment module, a hierarchical processing module and a data storage module; the data acquisition module is used for collecting multi-source heterogeneous original data of a power enterprise; the hierarchical judgment module is used for extracting attribute features of the multi-source heterogeneous original data, matching the attribute features with preset hierarchical rules and calculating to obtain corresponding data hierarchical results; the hierarchical processing module is used for allocating corresponding encryption strength configurations, transmission priority allocations and access permission controls to the data based on the data hierarchical results to obtain hierarchical processing data; and the data storage module is used for differentiating storage of the hierarchical processing data based on the data hierarchical results of the hierarchical processing data; the application can accurately quantize hierarchical judgment and accurately allocate corresponding differentiated processing actions and storage actions to the data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a data hierarchical processing system and method. Background Technology

[0002] Against the backdrop of accelerated digital transformation, data has become a core production factor for enterprises, widely used in key scenarios such as business decision-making, product optimization, and risk management. However, the current data ecosystem is characterized by "diversified sources, heterogeneous structures, and massive scale." Different types of data have vastly different value densities, sensitivity levels, and timeliness requirements. For example, financial transaction data and personal identity information are considered highly sensitive and valuable data, while redundant system log records and temporary interaction data are considered low-value data.

[0003] Existing data grading technologies are ill-suited to the needs of the complex data ecosystem described above, and suffer from several problems: First, existing data grading solutions rely on static rules or manual annotation, which cannot comprehensively evaluate the multi-dimensional attribute features of dynamically extracted data and lack a precise quantitative system for grading determination, resulting in large deviations in grading results; Second, existing post-grading differentiated processing strategies either over-invest in resources, leading to waste, or lack sufficient protection, resulting in the risk of leakage of high-value data, lacking a basis for post-grading differentiated processing and not adapting to the post-grading results.

[0004] Therefore, how to accurately quantify, classify, and differentiate data is an important issue that urgently needs to be addressed. Summary of the Invention

[0005] To address the problem that existing technologies cannot accurately quantify, classify, and differentiate data, this invention proposes a data classification and processing system, comprising: Data acquisition module, hierarchical judgment module, hierarchical processing module, and data storage module; The data acquisition module is used to collect multi-source heterogeneous raw data from power companies; The classification determination module is used to extract the attribute features of the multi-source heterogeneous raw data, match and calculate the attribute features with the preset classification rules, and obtain the corresponding data classification results. The hierarchical processing module is used to allocate corresponding encryption strength configuration, transmission priority allocation and access control to the data based on the data hierarchical results, so as to obtain hierarchically processed data. The data storage module is used to perform differentiated storage of the graded processed data based on the data grading results of the graded processed data.

[0006] Optionally, the hierarchical determination module includes: a rule storage unit, a feature extraction unit, and a matching determination unit; The rule storage unit is used to store preset hierarchical rules, which include, but are not limited to: scene dimension rules, scoring mapping rules, weight parameter rules, and hierarchical threshold rules. The feature extraction unit is used to extract attribute features from the multi-source heterogeneous raw data. The attribute features include, but are not limited to: data source, data size, data correlation, data access frequency, and data sensitivity. The matching determination unit is used to match the attribute features sequentially with the scene dimension rule, scoring mapping rule and weight parameter rule in the preset grading rules to obtain a matching result; calculate the grading score of the data based on the matching result, and match the grading score of the data with the grading threshold rule in the preset grading rules to obtain the corresponding data grading result; The data classification results include one of the following: core-level data, important-level data, and general-level data.

[0007] Optionally, the formula for calculating the graded score is:

[0008] in, To assign a graded score to the data; The number of attribute features, and the value of the number is based on the scene dimension rule of the preset hierarchical rule; The weight is the weight of the k-th attribute feature, and the value of the weight is based on the weight parameter rule of the preset classification rule; The normalized score is the score of the k-th attribute feature, and the normalized score is based on the scoring mapping rule of the preset grading rule.

[0009] Optionally, the data storage module is specifically used for: When the data classification result of the hierarchical processing data is core-level data, the core-level data is stored on an encrypted cloud server or a local physically isolated storage device using a multi-replica backup mechanism. When the data classification result of the graded processing data is important data, the important data is stored in an encrypted cloud storage node using a dual-copy backup mechanism; When the data classification result of the graded processing data is general-level data, the general-level data is stored in a general cloud storage server using a single-copy conventional storage mechanism. The calculation formula for the multi-replica backup mechanism is as follows:

[0010] in, The number of copies of core-level data that need to be backed up; For the reliability requirements of core-level data, the value range is [0.999, 1]. For the reliability of a single storage device; This is the floor function.

[0011] Optionally, the system further includes: a data preprocessing module; The data preprocessing module is used to clean, deduplicatize, and standardize the received multi-source heterogeneous raw data to obtain preprocessed data. The cleaning process includes: removing invalid values ​​with format errors and values ​​outside the preset range, as well as outliers and missing values ​​identified by the Z-Score method.

[0012] Optionally, the system further includes: a data monitoring module; The data monitoring module is used to record the operating status data of the data acquisition module, the data preprocessing module, the hierarchical judgment module and the hierarchical processing module as a data processing log, and generate corresponding alarm information and send it to the management terminal when an abnormal state is detected in the operating status data.

[0013] Optionally, the abnormal state includes at least one of the following: data acquisition interruption, preprocessing failure rate exceeding a preset threshold, graded judgment time exceeding a preset duration, and data storage capacity less than a preset capacity threshold; The formula for calculating the preprocessing failure rate is as follows:

[0014] in, To account for the preprocessing failure rate, This refers to the number of data samples that failed to be preprocessed in the data preprocessing module. This refers to the total number of data samples participating in the preprocessing in the data preprocessing module.

[0015] Optionally, the hierarchical processing module is specifically used for: For core-level data, the national cryptographic algorithm SM4 is used for full encryption, core-level transmission priority is assigned, and administrator-only access permissions are configured. For critical data, the Advanced Encryption Standard (AES-256) algorithm is used for encryption, a critical transmission priority is assigned, and departmental-level access permissions are configured. For general-level data, use a standard lightweight encryption algorithm for encryption and configure access permissions at the job level or for all employees in the department. The formula for calculating the transmission priority is:

[0016]

[0017] in, This is a quantified value representing the transmission priority of data within the same data level. This refers to the rank weight coefficient of data within the same rank. This is the timeliness weighting coefficient for data within the same data level; For data gradation values, core-level data Importance level: 3 2, general level data =1; This is a quantitative value for the timeliness of data within the same data level.

[0018] Furthermore, this application also provides a data hierarchical processing method, including the following steps: Collect multi-source heterogeneous raw data from power companies; The attribute features of the multi-source heterogeneous raw data are extracted, and the attribute features are matched and calculated with preset classification rules to obtain the corresponding data classification results. Based on the data classification results, corresponding encryption strength configurations, transmission priority allocations, and access control are assigned to the data to obtain classified data processing. Based on the data classification results of the classified data, the classified data is stored differentially.

[0019] Optionally, the step of extracting attribute features from the multi-source heterogeneous raw data, matching and calculating the attribute features with preset classification rules to obtain corresponding data classification results includes: The attribute features of the multi-source heterogeneous raw data are extracted; the attribute features include, but are not limited to, data source, data size, data correlation, data access frequency, and data sensitivity; the attribute features are sequentially matched with the scene dimension rules, scoring mapping rules, and weight parameter rules in the preset grading rules to obtain matching results; the grading score of the data is calculated based on the matching results, and the grading score of the data is matched with the grading threshold rules in the preset grading rules to obtain the corresponding data grading results; the data grading results include one of the following: core-level data, important-level data, and general-level data.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a data classification processing system and method, comprising: a classification determination module matching and calculating the attribute characteristics of multi-source heterogeneous raw data with preset classification rules to obtain corresponding data classification results; a classification processing module assigning corresponding encryption strength configurations, transmission priority allocations, and access control to the data based on the data classification results; and a data storage module using the data classification results to perform differentiated storage of the data processed by the classification processing module. In this invention, the classification determination module, through matching and calculation, can accurately quantify and classify multi-source heterogeneous raw data; subsequently, the classification processing module and the data storage module can use the data classification results as the basis for differentiated processing, accurately assigning corresponding differentiated processing actions and storage actions to the data. Attached Figure Description

[0021] Figure 1 This is a structural diagram of a data hierarchical processing system according to the present invention; Figure 2 This is a framework diagram of a data hierarchical processing system according to the present invention; Figure 3 This is a flowchart of a data hierarchical processing method according to the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device according to the present invention. Detailed Implementation

[0022] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0023] Example 1: This invention provides a data hierarchical processing system, such as... Figure 1 The system consists of a data acquisition module, a classification judgment module, a classification processing module, and a data storage module.

[0024] The data acquisition module is used to collect multi-source heterogeneous raw data from power companies.

[0025] The classification and determination module is used to extract attribute features from multi-source heterogeneous raw data, match and calculate the attribute features with preset classification rules, and obtain the corresponding data classification results.

[0026] The hierarchical processing module is used to assign corresponding encryption strength configurations, transmission priority allocations, and access control to data based on the data hierarchical results, thereby obtaining hierarchically processed data.

[0027] The data storage module is used to perform differentiated storage of data based on the data classification results of the hierarchical processing.

[0028] The following is combined with Figure 2The framework diagram of the data hierarchical processing system provides a detailed explanation of the embodiments of the present invention. In one example of this embodiment, the data acquisition module includes multiple acquisition units and a data aggregation unit. Each acquisition unit is adapted to a data source type and is used to acquire raw data of the corresponding type. The data aggregation unit is used to receive multi-source heterogeneous raw data acquired by each acquisition unit, and after adding data source identifiers to the multi-source heterogeneous raw data, it transmits it to the data preprocessing module for preprocessing.

[0029] In this embodiment, the classification determination module includes a rule storage unit, a feature extraction unit, and a matching determination unit. The rule storage unit stores preset classification rules. The feature extraction unit extracts attribute features from the multi-source heterogeneous raw data. The matching determination unit sequentially matches the attribute features with the scene dimension rules, scoring mapping rules, and weight parameter rules in the preset classification rules to obtain matching results; it calculates the classification score of the data based on the matching results, and matches the data's classification score with the classification threshold rules in the preset classification rules to obtain the corresponding data classification result.

[0030] In one example, the preset grading rules include, but are not limited to: scenario dimension rules, scoring mapping rules, weight parameter rules, and grading threshold rules; in another example, the preset grading rules include, but are not limited to: data sensitivity threshold rules and data timeliness requirement rules.

[0031] In one example, the attribute characteristics include, but are not limited to: data source, data size, data correlation, data access frequency, and data sensitivity; the data classification results may include one of the following: core-level data, important-level data, and general-level data.

[0032] The formula for calculating the graded score is expressed as follows:

[0033] in, To assign a graded score to the data; The number of attribute features, and the value of the number is based on the scene dimension rule of the preset hierarchical rule; The weight of the k-th attribute feature (satisfying) Furthermore, the weight values ​​are based on the weight parameter rules of the preset grading rules; This represents the normalized score of the k-th attribute feature, based on a pre-defined scoring mapping rule. The formula for calculating the grading score comprehensively evaluates the data value by considering multiple attribute features, and uses weight allocation to reflect the importance of different features, ensuring the objectivity and accuracy of the grading results.

[0034] In one example, the matching content of the matching decision unit includes, but is not limited to, matching of scene dimension rules, matching of scoring mapping rules, matching of weight parameter rules, and matching of grading threshold rules. For example, the detailed matching method for each match is as follows: 1. Matching of Scene-Dimensional Rules: Based on the attribute characteristics of the acquired multi-source heterogeneous raw data, match them with scene-dimensional rules to select features that can be included in this grading and scoring. For example, the official scenario rule in the scene-dimensional rules needs to include data sensitivity as a core feature, and the internet scenario rule needs to include data access frequency ...

[0035] 2. Matching of scoring mapping rules: Based on the feature values ​​corresponding to the selected features for this grading, the feature values ​​are converted into normalized scores within the [0,1] interval according to the scoring mapping rules in the preset grading rules. In the scoring mapping rules, specific piecewise mapping functions or thresholds are defined. For example, the scoring mapping rules define that the higher the data sensitivity, the higher the threshold. The closer the value is to 1; or the higher the access frequency, The closer the value is to 1.

[0036] 3. Matching of weight parameter rules: Based on the business scenario in the matching scenario dimension rules, the weight coefficients of the features defined in the weight parameter rules are used. For example, the weight of data sensitivity in the power sector is 0.5.

[0037] 4. Matching of grading threshold rules: After selecting the above parameters, substitute them into the grading score calculation formula to obtain the grading score of the data. ,Will The data is matched against grading threshold rules to determine the grading result. For example: Data with a value greater than or equal to 0.8 is considered core-level data.

[0038] The grading determination module in this embodiment can improve the accuracy of grading determination. By extracting multi-dimensional attribute features of the data, it uses the weighted summation quantitative scoring model in the above example to calculate the grading score and realize the grading determination. This avoids the subjectivity of static rules and manual annotation, and significantly improves the accuracy and objectivity of the grading results of the three types of data: core data, important data, and general data, providing a reliable basis for subsequent differentiated processing.

[0039] In this embodiment, the data storage module is specifically used for: when the data classification result of the hierarchical processing is core-level data, using a multi-replica backup mechanism to store the core-level data on an encrypted cloud server or a local physically isolated storage device; when the data classification result of the hierarchical processing is important-level data, using a dual-replica backup mechanism to store the important-level data on an encrypted cloud storage node; and when the data classification result of the hierarchical processing is general-level data, using a single-replica conventional storage mechanism to store the general-level data on a general-purpose cloud storage server. For example, the calculation formula for the multi-replica backup mechanism is expressed as:

[0040] in, The number of copies of core-level data that need to be backed up; For the reliability requirements of core-level data, the value range is [0.999, 1]. For the reliability of a single storage device; This is a rounding function. The above formula can scientifically determine the number of backup copies based on the reliability requirements of core-level data and the performance of a single storage device, ensuring data reliability while avoiding the waste of storage resources caused by excessive backups.

[0041] The data hierarchical processing system of this embodiment may further include: a data preprocessing module, used to clean, deduplicate and standardize the received multi-source heterogeneous raw data to obtain preprocessed data.

[0042] In one example, the cleaning process includes: removing malformed values ​​and invalid values ​​outside a preset range, as well as outliers and missing values ​​identified using the Z-Score method. The Z-Score is calculated using the following formula:

[0043] in, Let Z be the Z-score value of the i-th data sample. Let be the original value of the i-th data sample. This is the mean of the data sample. This represents the standard deviation of the data sample. The above formula quantifies the dispersion of the data by calculating the degree to which the data sample deviates from the mean. If the value exceeds a preset threshold, the data sample is determined to be an outlier.

[0044] The data tiered processing system in this embodiment may further include a data monitoring module, used to record the operational status data of the data acquisition module, data preprocessing module, tiered judgment module, and tiered processing module as a data processing log, and to generate corresponding alarm information and send it to the management terminal when abnormal status data is detected. The data monitoring module enables a monitoring mechanism throughout the entire system process, allowing real-time awareness of abnormal statuses at each stage of tiered processing, facilitating rapid problem location and resolution. Furthermore, by employing high-strength encryption and strict access control for highly sensitive and high-value data, and a lightweight processing strategy for low-value data, the data monitoring module not only meets compliance requirements for data security protection but also improves overall data processing efficiency, helping enterprises fully unleash the value of their data.

[0045] In one example, abnormal states include, but are not limited to, at least one of the following: data acquisition interruption, preprocessing failure rate exceeding a preset threshold, grading judgment time exceeding a preset duration, and data storage capacity less than a preset capacity threshold. Alarm information includes, but is not limited to, at least one of the following: abnormal occurrence time, abnormal stage, and abnormal type. The formula for calculating the preprocessing failure rate is:

[0046] in, To account for the preprocessing failure rate, This represents the number of data samples that failed to be preprocessed in the data preprocessing module. This represents the total number of data samples participating in the preprocessing phase of the data preprocessing module. The above formula quantifies the degree of failure in the data preprocessing stage. ( When the preset failure rate threshold is reached, an abnormal alarm is triggered to remind managers to investigate the problem in a timely manner.

[0047] In this embodiment, the hierarchical processing module is specifically used for: core-level data, which can be fully encrypted using the national cryptographic algorithm SM4, can be assigned a core-level transmission priority, and can be configured with administrator-only access permissions; important-level data, which can be encrypted using the advanced encryption standard algorithm AES-256, can be assigned an important-level transmission priority, and can be configured with department-level access permissions; and general-level data, which can be encrypted using a conventional lightweight encryption algorithm, and can be configured with job-level or department-wide access permissions.

[0048] In one example, the formula for calculating transmission priority is:

[0049]

[0050] in, This is a quantified value representing the transmission priority of data within the same data level. This refers to the rank weight coefficient of data within the same rank. The timeliness weighting coefficient for data at the same level (satisfying) ); For data gradation values, core-level data Importance level: 3 2, general level data =1; This is a quantified value for the timeliness of data within the same data level (range [0,1], with higher values ​​for greater timeliness). The formula for calculating transmission priority can determine the priority order of data transmission through quantification, ensuring that high-level, high-timeliness data is transmitted first, thus improving the overall efficiency of data processing.

[0051] The data classification in the hierarchical judgment module, dividing the data into core-level, important-level, and general-level data, is only a coarse-grained priority classification. Further refinement, configurability, and multi-dimensional priority scheduling are needed through the transmission priority calculation formula. The specific formula for transmission priority is analyzed below: It provides a quantitative value for the timeliness of data within the same level, enabling continuous quantitative scoring of data at the same level. It can further distinguish the order of transmission within the same level, improve the accuracy of scheduling, and achieve fine sorting within the same level.

[0052] The rank weight coefficient of data within the same rank. This refers to the timeliness weighting coefficient for data within the same data level. These two factors can be differentiated to balance the weighting of data security level and business timeliness. In one example, the power production scenario prioritizes security level and can be configured... =0.8, =0.2, prioritizing the level; for internet e-commerce scenarios, efficiency and timeliness are both considered, and it is configurable. =0.5, =0.5, the two have equal weights; only by adjusting and This eliminates the need to modify the overall hierarchical architecture, allowing for flexible adaptation to scheduling strategies for different business scenarios.

[0053] This quantifies the transmission priority of data within the same data level, transforming the priority from a qualitative description into computable, continuous data. This facilitates the automatic execution of queue sorting and resource allocation by the transmission scheduling system, avoiding the subjectivity and lag of manual judgment, ensuring the standardized and automated operation of transmission scheduling, and thus providing a quantitative basis for automated scheduling.

[0054] The formula for transmission priority means that priority is no longer determined solely by data security level, but also incorporates the dimension of business timeliness. While ensuring high-level data security priority, it also takes into account the real-time needs of business, balancing data security and business efficiency, and achieving the effect of taking into account both security and business attributes.

[0055] The data classification processing system in this embodiment may further include: a rule update unit, used to dynamically adjust the parameter thresholds of the preset classification rules according to user feedback and data processing effects, so as to realize the adaptive optimization of the classification rules, which can flexibly adapt to changes in different business scenarios and dynamic update requirements of data attributes, thus expanding the applicability of the system.

[0056] In one example, the dynamic adjustment of the rule update unit is based on the configurable parameters in the preset hierarchical rules (such as scene dimension rules, scoring mapping rules, weight parameter rules, and hierarchical threshold rules). It is achieved through a closed-loop mechanism of "data collection - indicator evaluation - parameter optimization - gray-scale verification - rule activation". Specific adjustment methods may include: adjustments based on user feedback and adjustments based on data processing effects.

[0057] 1. Adjustments based on user feedback.

[0058] User feedback primarily refers to manual corrections and annotations of data grading results, as well as business requests. The adjustment logic includes: First, the system collects records of manual corrections made by users to the data grading results (e.g., data judged as important by the system is manually upgraded to core level or downgraded to general level), and statistically analyzes the frequency, direction, and magnitude of corrections for similar data to obtain similar correction samples. Next, when the number of similar correction samples reaches a preset trigger threshold, the system automatically attributes the cause and adjusts the corresponding rules. For example, if data corresponding to a certain feature is frequently upgraded in grading level, the weight of that feature in the weight parameter rule is increased accordingly after attribution; if the correction rate for a certain data level range is too high, the grading threshold for that data level in the grading threshold rule is adjusted accordingly after attribution.

[0059] In one example, users frequently change data of the "user identity information" category from important to core level. The system can automatically attribute this data to increase the weight of the "data sensitivity" feature or lower the classification threshold of core level data, so that subsequent data of the same category can be automatically identified as core level data.

[0060] 2. Adjustments based on data processing results.

[0061] Data processing effectiveness refers to quantifiable objective indicators during system operation. The adjustment logic includes: First, during system operation, the system continuously collects operational indicators throughout the entire process, including but not limited to: graded accuracy rate, core data leakage risk rate, transmission latency compliance rate, storage resource utilization rate, or data access hit rate. Second, when operational indicators deviate from the preset target range, the system automatically triggers rule parameter optimization actions.

[0062] In one example, parameter optimization actions include: if the proportion of core-level data is too high and the storage resource utilization exceeds the standard, it indicates that the classification threshold is too loose, and the scoring threshold of the core level can be increased to narrow the range of core-level data; if in the data transmission priority allocation results, high-time-sensitivity data frequently exceeds the transmission delay standard, the timeliness weight coefficient in the transmission priority formula can be increased, or the scoring mapping rules of data timeliness characteristics can be adjusted to improve the priority score of high-time-sensitivity data; if the classification result has a low degree of matching with business access, or high-frequency access data is classified into a low level, the feature weight of data access frequency can be increased.

[0063] The data hierarchical processing system in this embodiment may further include: an interface extension module, which provides standardized data input / output interfaces, supports connection with external business systems, data platforms and third-party analysis tools, and realizes bidirectional data interaction and sharing.

[0064] In one example, the data classification processing system of this embodiment is used to classify power industry data. The specific processing procedure is as follows: First, it is clear that the data hierarchical processing system in this embodiment is specifically applied to the production and operation data processing scenario of power enterprises. The data types involved include core data of power grid dispatch, electricity consumption information data of power users, and operation and maintenance monitoring data of power equipment. It is necessary to focus on ensuring the security and real-time performance of power grid dispatch data and adapt to the dual needs of data security and production and operation in the power industry.

[0065] For the data acquisition module: three dedicated acquisition units are set up to adapt to the power grid dispatching system, the electricity consumption information acquisition system, and the equipment operation and maintenance monitoring platform, respectively. The data aggregation unit adds data source identifiers such as "dispatch-XX", "electricity consumption-XX" and "operation and maintenance-XX" to the collected data before transmitting them to the preprocessing module.

[0066] For the data preprocessing module: the Z-score method is used to identify outliers. It needs to be greater than the preset threshold of 3, that is >3; Remove incorrectly formatted instruction data from the scheduling data and invalid collected data from the electricity consumption information, standardize the format of the operation and maintenance monitoring data, and unify the data collection timestamp format.

[0067] For the grading determination module: In the preset grading rules' weight parameters, the data value weights are configured as follows: data sensitivity weight w1 = 0.5, data timeliness weight w2 = 0.3, and data relevance weight w3 = 0.2. The grading score is calculated using a weighted summation formula, with the grading thresholds set as follows: S ≥ 0.8 for core-level data, 0.5 ≤ S < 0.8 for important-level data, and S < 0.5 for general-level data. The final determination result is: core data for power grid dispatch is core-level data, electricity user consumption information data is important-level data, and power equipment operation and maintenance monitoring data is general-level data.

[0068] For the hierarchical processing module: core-level data is fully encrypted using the national cryptographic SM4 algorithm, with α=0.8 and β=0.2 in the transmission priority quantization formula. =3. Timeliness =0.95, calculated as follows =0.8×3+0.2×0.95=2.59 (highest priority), access is only granted to the dispatch center administrator; important data is encrypted using the AES-256 algorithm. =2、 =0.8, =0.8×2+0.2×0.8=1.76 (second highest priority), access is granted to the marketing and electricity management departments; general-level data uses conventional lightweight encryption. =1、 =0.6, =0.8×1+0.2×0.6=0.92 (normal priority), open to all staff of the equipment operation and maintenance department.

[0069] For the data storage module and data monitoring module: core-level data is stored on a locally physically isolated encrypted storage device, and the number of backup copies is determined according to the formula. =0.9999 (Power dispatch level reliability requirement) =0.99, calculated as follows Configure dual-replica off-site backup; preset the preprocessing failure rate threshold for the data monitoring module. =4%, focusing on monitoring the power grid dispatch data preprocessing stage. When the failure rate exceeds the threshold, an alarm message is immediately sent to the power operation and maintenance management terminal.

[0070] In one example, the data classification processing system of this embodiment is used to classify and process official data. The specific processing procedure is as follows: First, it should be clarified that the data hierarchical processing system in this embodiment is specifically applied to the government data management platform. The data types involved include resident information, government approval process data, public service disclosure data, and government office notice data. It is necessary to balance data security and public service efficiency, and adapt to the compliance requirements for government data sharing.

[0071] For the data acquisition module: the acquisition unit is set to adapt to the resident management system, the government approval system, the government publicity platform and the internal OA system. The data aggregation unit transmits data after adding the government department's identification.

[0072] For the data preprocessing module: the Z-score method is used ( >3) Identify abnormal processing time data in the approval data, remove incorrectly formatted ID number data in the resident information, and standardize the format of the publicized data (unify it to PDF format).

[0073] For the grading and determination module: the data value weight is configured as: data sensitivity. =0.5, data timeliness =0.2, data correlation =0.2, data size =0.1; The grading and scoring formula is the same as the grading and processing of power industry data in the previous example. Grading thresholds: S≥0.75 is core level, 0.5≤S<0.75 is important level, and S<0.5 is general level. Judgment results: Resident information is core level data, government approval process data is important level data, public service disclosure data is ordinary level data, and government office notice data is low value level data.

[0074] For the hierarchical processing module: core-level data is encrypted using the national standard SM4, with transmission priorities α=0.6 and β=0.4. =3、 =0.8, =0.6×3+0.4×0.8=2.12; Access permissions are only granted to designated personnel in the government data management department; Important data is encrypted using AES-256. =2、 =0.9, =0.6×2+0.4×0.9=1.92; Access permissions are granted to personnel in departments related to official approvals; General-level data uses standard lightweight encryption. =1、 =0.5, =0.6×1+0.4×0.5=0.8, which is open to all staff members of the civil service system.

[0075] For the data storage module and data monitoring module: core-level data is stored on an encrypted cloud server. =0.9995、 =0.98, Dual-copy backup; critical data is stored on encrypted cloud storage nodes with single-copy backup; general data is stored on general-purpose cloud storage servers with regular single-copy storage; the monitoring module has a preset preprocessing failure rate threshold. =3%, focusing on monitoring the preprocessing stage of resident information, and alerting the government data security management terminal when abnormalities occur.

[0076] In one example, the data classification processing system of this embodiment is used to classify and process the data of Internet companies. The specific processing procedure is as follows: First, it should be clarified that the data hierarchical processing system in this embodiment is specifically applied to internet e-commerce enterprises. The data types involved include user order data, user browsing behavior data, product information data, and platform advertising data. It is necessary to balance data processing efficiency and storage costs to support the needs of precision marketing.

[0077] For the data acquisition module: the acquisition unit is adapted to e-commerce transaction systems, user behavior analysis systems, product management systems and advertising systems, and the data aggregation unit adds a timestamp identifier for data generation.

[0078] For the data preprocessing module: Z-score method ( >3) Identify abnormal order amount data, remove duplicate user browsing records, and standardize the fields of product information data (unify product classification codes and price units).

[0079] For the hierarchical judgment module: the data value weight is configured as: data access frequency. =0.3, data correlation =0.25, data timeliness =0.25, data size =0.2; The grading formula is the same as before, and the grading thresholds are: S≥0.8 for core level, 0.5≤S<0.8 for important level, and S<0.5 for general level. Judgment results: User order data is core level, user browsing behavior data and product information data are important level, and platform advertising data is low-value level.

[0080] For the hierarchical processing module: core-level data is encrypted with SM4, and transmission priority α=0.5, β=0.5. =3、 =0.95, =0.5×3+0.5×0.95=1.975; Access permissions are granted to core personnel in the operations and finance departments; Important data is encrypted with AES-256. =2、 =0.8, =0.5×2+0.5×0.8=1.4; Access permissions are granted to operations and marketing personnel; general-level data uses standard lightweight encryption. =1、 =0.4, =0.5×1+0.5×0.4=0.7, which is open to all members of the advertising operations department.

[0081] For the data storage module and data monitoring module: core-level data is stored on an encrypted cloud server. =0.999、 =0.99, Single copy + off-site backup; critical data is stored on encrypted cloud storage nodes with standard single copy storage; general data is stored on general-purpose cloud storage servers with low-cost distributed storage; the monitoring module has a preset preprocessing failure rate threshold. It focuses on monitoring the order data processing stage to ensure the integrity of transaction data.

[0082] Based on three examples of data classification and processing systems for power industry data, government data, and internet company data, a comparison is made from four dimensions: application scenario adaptability, core parameter configuration, performance, and cost investment. Table 1 shows the comparative analysis of data applications. Table 1 Comparative Analysis of Data Application

[0083] The above comparison results demonstrate that by flexibly adjusting the weights of the classification judgment module, the parameters for transmission priority calculation, and the parameters for backup reliability, data can be divided into core, important, and general levels, accurately adapting to the application scenarios of different industries such as the power industry, government, and internet companies. Specifically, the power industry prioritizes high security and low latency for core scheduling data, configuring high data level weights and the highest reliability backups; government scenarios balance security and service efficiency, increasing the weight of timeliness; and internet companies prioritize a balance between cost and efficiency, adopting equalized priority weights and a lightweight backup strategy, verifying the universality and adaptability advantages of this invention.

[0084] Example 2: Based on the same inventive concept, embodiments of the present invention also provide a data hierarchical processing method, such as... Figure 3 The flowchart shown includes the following steps: Step 1: Collect multi-source heterogeneous raw data from power companies.

[0085] Step 2: Extract the attribute features of the multi-source heterogeneous raw data, match and calculate the attribute features with the preset classification rules, and obtain the corresponding data classification results.

[0086] In step two, attribute features of the multi-source heterogeneous raw data are extracted. Optional attribute features include, but are not limited to: data source, data size, data correlation, data access frequency, and data sensitivity.

[0087] Next, the attribute features are matched sequentially with the scene dimension rules, scoring mapping rules, and weight parameter rules in the preset grading rules to obtain matching results. Based on the matching results, the data grading score is calculated, and the data grading score is matched with the grading threshold rules in the preset grading rules to obtain the corresponding data grading results. Among them, the optional data grading results include: core-level data, important-level data, and general-level data.

[0088] Step 3: Based on the data classification results, assign corresponding encryption strength configurations, transmission priority allocations, and access control to the data to obtain classified data processing.

[0089] Step 4: Based on the data classification results of the hierarchical processing, perform differentiated storage of the hierarchical processing data.

[0090] Example 3: Based on the same inventive concept, the present invention also provides a computer device, such as... Figure 4 As shown, the computer device includes a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, and is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function, so as to implement the steps of a data hierarchical processing method in the above embodiment.

[0091] Example 4: Based on the same inventive concept, this invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the steps of a data hierarchical processing method in the above embodiments.

[0092] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. A data hierarchical processing system, characterized in that, include: Data acquisition module, hierarchical judgment module, hierarchical processing module, and data storage module; The data acquisition module is used to collect multi-source heterogeneous raw data from power companies; The classification determination module is used to extract the attribute features of the multi-source heterogeneous raw data, match and calculate the attribute features with the preset classification rules, and obtain the corresponding data classification results. The hierarchical processing module is used to allocate corresponding encryption strength configuration, transmission priority allocation and access control to the data based on the data hierarchical results, so as to obtain hierarchically processed data. The data storage module is used to perform differentiated storage of the graded processed data based on the data grading results of the graded processed data.

2. The system as described in claim 1, characterized in that, The hierarchical determination module includes: a rule storage unit, a feature extraction unit, and a matching determination unit; The rule storage unit is used to store preset hierarchical rules, which include, but are not limited to: scene dimension rules, scoring mapping rules, weight parameter rules, and hierarchical threshold rules. The feature extraction unit is used to extract attribute features from the multi-source heterogeneous raw data. The attribute features include, but are not limited to: data source, data size, data correlation, data access frequency, and data sensitivity. The matching determination unit is used to match the attribute features sequentially with the scene dimension rule, scoring mapping rule and weight parameter rule in the preset grading rules to obtain a matching result; calculate the grading score of the data based on the matching result, and match the grading score of the data with the grading threshold rule in the preset grading rules to obtain the corresponding data grading result; The data classification results include one of the following: core-level data, important-level data, and general-level data.

3. The system as described in claim 2, characterized in that, The formula for calculating the graded score is as follows: in, To assign a graded score to the data; The number of attribute features, and the value of the number is based on the scene dimension rule of the preset hierarchical rule; The weight is the weight of the k-th attribute feature, and the value of the weight is based on the weight parameter rule of the preset classification rule; The normalized score is the score of the k-th attribute feature, and the normalized score is based on the scoring mapping rule of the preset grading rule.

4. The system as described in claim 2, characterized in that, The data storage module is specifically used for: When the data classification result of the graded processing data is core-level data, the core-level data is stored on an encrypted cloud server or a local physically isolated storage device using a multi-replica backup mechanism. When the data classification result of the graded processing data is important data, the important data is stored in an encrypted cloud storage node using a dual-copy backup mechanism; When the data classification result of the graded processing data is general-level data, the general-level data is stored in a general cloud storage server using a single-copy conventional storage mechanism. The calculation formula for the multi-replica backup mechanism is as follows: in, The number of copies of core-level data that need to be backed up; For the reliability requirements of core-level data, the value range is [0.999, 1]. For the reliability of a single storage device; This is the floor function.

5. The system as described in claim 1, characterized in that, Also includes: Data preprocessing module; The data preprocessing module is used to clean, deduplicatize, and standardize the received multi-source heterogeneous raw data to obtain preprocessed data. The cleaning process includes: removing invalid values ​​with format errors and values ​​outside the preset range, as well as outliers and missing values ​​identified by the Z-Score method.

6. The system as described in claim 5, characterized in that, Also includes: Data monitoring module; The data monitoring module is used to record the operating status data of the data acquisition module, the data preprocessing module, the hierarchical judgment module and the hierarchical processing module as a data processing log, and generate corresponding alarm information and send it to the management terminal when an abnormal state is detected in the operating status data.

7. The system as described in claim 6, characterized in that, The abnormal states include at least one of the following: data acquisition interruption, preprocessing failure rate exceeding a preset threshold, graded judgment time exceeding a preset duration, and data storage capacity less than a preset capacity threshold. The formula for calculating the preprocessing failure rate is as follows: in, To account for the preprocessing failure rate, This refers to the number of data samples that failed to be preprocessed in the data preprocessing module. This refers to the total number of data samples participating in the preprocessing in the data preprocessing module.

8. The system as described in claim 2, characterized in that, The hierarchical processing module is specifically used for: For core-level data, the national cryptographic algorithm SM4 is used for full encryption, core-level transmission priority is assigned, and administrator-only access permissions are configured. For critical data, the Advanced Encryption Standard (AES-256) algorithm is used for encryption, a critical transmission priority is assigned, and departmental-level access permissions are configured. For general-level data, use a standard lightweight encryption algorithm for encryption and configure access permissions at the job level or for all employees in the department. The formula for calculating the transmission priority is: in, This is a quantified value representing the transmission priority of data within the same data level. This refers to the rank weight coefficient of data within the same rank. This is the timeliness weighting coefficient for data within the same data level; For data gradation values, core-level data Importance level: 3 2, general level data =1; This is a quantitative value for the timeliness of data within the same data level.

9. A data hierarchical processing method, characterized in that, Includes the following steps: Collect multi-source heterogeneous raw data from power companies; The attribute features of the multi-source heterogeneous raw data are extracted, and the attribute features are matched and calculated with preset classification rules to obtain the corresponding data classification results. Based on the data classification results, corresponding encryption strength configurations, transmission priority allocations, and access control are assigned to the data to obtain classified data processing. Based on the data classification results of the classified data, the classified data is stored differentially.

10. The method as described in claim 9, characterized in that, The step of extracting attribute features from the multi-source heterogeneous raw data, matching and calculating the attribute features with preset classification rules to obtain corresponding data classification results includes: The attribute features of the multi-source heterogeneous raw data are extracted; the attribute features include, but are not limited to: data source, data size, data correlation, data access frequency, and data sensitivity. The attribute features are sequentially matched with the scene dimension rules, scoring mapping rules, and weight parameter rules in the preset grading rules to obtain matching results; the grading score of the data is calculated based on the matching results, and the grading score of the data is matched with the grading threshold rules in the preset grading rules to obtain the corresponding data grading results; The data classification results include one of the following: core-level data, important-level data, and general-level data.

Images