A dynamic data management method and system

Abstract

The present invention provides a dynamic data management method and system thereof. The present invention establishes a dynamic adjustment mechanism to assign weights to various types of data, and formulates corresponding processing and encryption strategies, thereby realizing dynamic management and security protection of data. High-priority and high-weight data are responded to immediately and processed efficiently, while low-priority data occupies fewer resources, thereby ensuring security without excessively consuming computing resources, thereby improving data processing efficiency and quality. At the same time, by setting conflict decision preset rules for data processing priority and encryption priority, the method and system effectively solve the priority conflict problem in the data processing process, ensure the efficiency and security of data processing, and provide good services for the processing of big data.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a dynamic data management method and system thereof. Background Art

[0002] In the current era of big data, enterprises and organizations are facing unprecedented challenges in data management and processing. With the explosive growth of data volumes, the diversification of data sources, and the complexity of data types, traditional static data management methods are no longer able to meet the growing business needs. For example, in the process of processing massive amounts of data, there is no effective mechanism to distinguish the importance and sensitivity of data, resulting in an irrational allocation of processing and storage resources between critical data and ordinary data. High-value data processing delays often occur, while a large amount of non-critical data occupies a large amount of computing and storage resources, affecting overall processing efficiency and cost-effectiveness. In real-time data stream processing scenarios, the processing requirements and urgency of different types of data vary, but there is a lack of effective mechanisms to dynamically adjust processing priorities. Especially when resources are limited, how to balance the timeliness of data processing and resource utilization efficiency has become an urgent problem to be solved. In data processing, there are also problems with processing priority conflicts and insufficient dynamic adaptability.

[0003] Therefore, it is necessary to provide a dynamic data management method and system thereof to solve the above technical problems. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a dynamic data management method and system for solving the problems of uneven resource allocation, security compliance challenges, priority conflict processing and insufficient dynamic adaptability required for the management of dynamic data in a big data environment.

[0005] The present invention provides a dynamic data management method, which includes the following steps:

[0006] S1. Clarify the needs of dynamic data management and conduct dynamic data collection based on big data;

[0007] S2. Preprocess data from different sources, merge the preprocessed data from different sources using data matching and association technology to form a unified data set, and store the data set;

[0008] S3. Classify the data in the dataset and determine its weight according to its importance and sensitivity. Develop data weight calculation rules for each type of classified data.

[0009] S4. Based on the data weight calculation rules, establish a dynamic adjustment mechanism to assign weights to various types of data, and formulate corresponding processing and encryption strategies;

[0010] S5. Based on the corresponding processing and encryption strategies, set the conflict decision-making preset rules for data processing priority and encryption priority;

[0011] S6. Allocate system resources for processing the weighted data according to the corresponding processing and encryption strategies and conflict resolution preset rules, analyze and encrypt the data, and obtain archived data.

[0012] S7. Develop access rights management policies and monitor data access based on archived data for use by internal applications, partners, or customers to ensure data security and user privacy.

[0013] Preferably, step S3 includes the following steps:

[0014] S3.1. Classify the data in the dataset based on its characteristics and usage to obtain various types of data. The data categories specifically include user data, transaction data, and log data.

[0015] S3.2. Evaluate the importance and sensitivity of each type of classified data based on business needs and security levels, and obtain evaluation results;

[0016] S3.3. Based on the evaluation results, establish specific rules and algorithms for data weight calculation to ensure that the final weight distribution is reasonable and reflects the actual situation.

[0017] Preferably, step S4 includes the following steps:

[0018] S4.1. Adjust data weight calculation rules regularly or in real time based on business changes and actual needs;

[0019] S4.2. Based on the adjusted data weight calculation rules, define clear trigger conditions such as the addition of new data sources, and establish a dynamic adjustment mechanism through adaptive algorithms to assign new weights to various types of data after the trigger conditions in real time;

[0020] S4.3. Develop processing and encryption strategies based on the weights assigned to each type of data.

[0021] Preferably, step S5 includes the following steps:

[0022] S5.1. Analyze possible resource conflicts between processing priority and encryption priority;

[0023] S5.2. Based on resource conflicts, define decision-making factors, including data importance, business impact, compliance requirements, and resource cost factors;

[0024] S5.3. Set conflict decision preset rules based on the defined decision elements, including priority rules, resource allocation rules, and emergency handling rules.

[0025] Preferably, step S6 includes the following steps:

[0026] S6.1. Allocate computing and storage resources to each type of data based on its weight and processing priority.

[0027] S6.2. Monitor resource usage to ensure that high-weight data receives sufficient resource support.

[0028] S6.3. When resources are insufficient or conflicts occur, real-time adjustments and allocations are made based on the preset conflict decision rules.

[0029] A dynamic data management system, comprising:

[0030] Data collection module, used to clarify the needs of dynamic data management and perform dynamic data collection based on big data;

[0031] The data integration module is used to pre-process data from different sources, merge the pre-processed data from different sources using data matching and association technology to form a unified data set, and store the data set;

[0032] The weighting module is used to classify the data in the dataset and determine the weights according to their importance and sensitivity. It also formulates data weight calculation rules for each type of classified data.

[0033] The weight assignment module is used to establish a dynamic adjustment mechanism to assign weights to various types of data based on data weight calculation rules, and formulate corresponding processing and encryption strategies;

[0034] A decision module is used to set conflict decision preset rules for data processing priority and encryption priority based on the formulation of corresponding processing and encryption strategies;

[0035] The allocation processing module is used to allocate system resources for processing various types of weighted data according to the corresponding processing and encryption strategies and conflict decision preset rules, and analyze, process and encrypt various types of data to obtain archived data;

[0036] The access module is used to formulate access rights management policies and monitor data access based on archived data for use by internal applications, partners, or customers to ensure data security and user privacy.

[0037] Preferably, the weight setting module includes:

[0038] The classification module is used to classify the data in the dataset according to its characteristics and usage, and obtain various types of classified data. The data categories specifically include user data, transaction data, and log data;

[0039] The evaluation module is used to evaluate the importance and sensitivity of various types of classified data based on business needs and security levels, and obtain evaluation results;

[0040] The rule setting module is used to establish specific rules and algorithms for data weight calculation rules based on the evaluation results to ensure that the final weight distribution is reasonable and reflects the actual situation.

[0041] Preferably, the weight assigning module includes:

[0042] Adjustment module, used to adjust data weight calculation rules regularly or in real time according to business changes and actual needs;

[0043] The dynamic identification module is used to calculate the data weight based on the adjusted data weight rules, define clear trigger conditions such as the addition of new data sources, and establish a dynamic adjustment mechanism through adaptive algorithms to assign new weights to various types of data after the trigger conditions in real time;

[0044] The policy formulation module is used to formulate processing policies and encryption policies based on the weights assigned to various types of data.

[0045] Preferably, the decision module includes:

[0046] An analysis module for analyzing possible resource conflicts between processing priority and encryption priority;

[0047] Define identification modules to define decision factors based on resource conflicts, including data importance, business impact, compliance requirements, and resource cost factors;

[0048] The setting module is used to set conflict decision preset rules based on the defined decision elements, including priority rules, resource allocation rules and emergency handling rules.

[0049] Preferably, the allocation processing module includes:

[0050] The first-level scheduling module is used to allocate computing and storage resources to various types of data based on the data's weight and processing priority;

[0051] The monitoring module is used to monitor the usage of computing resources and storage resources to ensure that high-weight data has sufficient resource support;

[0052] The secondary scheduling module is used to make real-time adjustments and allocations based on the preset conflict decision rules when resources are insufficient or conflicts occur.

[0053] Compared with related technologies, the dynamic data management method and system provided by the present invention have the following beneficial effects:

[0054] The present invention establishes a dynamic adjustment mechanism to assign weights to various types of data and formulates corresponding processing and encryption strategies, thereby realizing dynamic management and security protection of data. High-priority and high-weight data are responded to immediately and processed efficiently, while low-priority data takes up fewer resources, which ensures security without excessive consumption of computing resources, thereby improving data processing efficiency and quality. At the same time, by setting conflict decision preset rules for data processing priority and encryption priority, the method and system effectively solve the priority conflict problem in the data processing process, ensure the efficiency and security of data processing, and provide good services for the processing of big data. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A flow chart of a dynamic data management method of the present invention;

[0056] Figure 2 A block diagram of a dynamic data management system of the present invention;

[0057] Figure 3 A block diagram of a weight formulation module system for a dynamic data management system of the present invention;

[0058] Figure 4 A block diagram of a weight assignment module system for a dynamic data management system of the present invention;

[0059] Figure 5 A system block diagram of a decision module of a dynamic data management system of the present invention;

[0060] Figure 6 This is a system block diagram of an allocation processing module of a dynamic data management system of the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0062] Example 1

[0063] This embodiment provides a dynamic data management method, which includes the following steps:

[0064] S1. Clarify the needs of dynamic data management and collect dynamic data based on big data.

[0065] Specifically, it is necessary to clarify the goals, scope and specific needs of data management, and then formulate a data collection strategy based on these needs, including the frequency, data type, source, etc. of collection, and use big data collection tools or technologies to obtain data from various data sources in real time or in batches to ensure that the collected data is relevant, accurate and useful, providing a solid foundation for subsequent data processing and analysis.

[0066] S2. Preprocess the data from different sources, merge the preprocessed data from different sources using data matching and association technology to form a unified data set, and store the data set.

[0067] Specifically, data preprocessing includes steps such as cleaning, conversion, and format unification to eliminate errors, redundancies, and inconsistencies in the data. Then, data matching and association techniques are used to merge data from different sources to form a complete and unified data set. Finally, an appropriate storage solution is selected to store the data set efficiently and securely to eliminate data differences and conflicts and improve data consistency and accuracy.

[0068] S3. Classify the data in the dataset and determine the weight according to its importance and sensitivity. Develop data weight calculation rules for each type of classified data.

[0069] Specifically, the data in the dataset is classified according to its characteristics and usage, and then the importance and sensitivity of each type of data are evaluated, the corresponding weight is determined, and finally, weight calculation rules are formulated to provide a clear basis for weight assignment for different types of data, so as to have a clearer understanding of the value and risks of each type of data, and provide an important basis for subsequent data processing and encryption strategy formulation.

[0070] S4. Based on the data weight calculation rules, establish a dynamic adjustment mechanism to assign weights to various types of data, and formulate corresponding processing and encryption strategies.

[0071] Specifically, according to the data weight calculation rules, the data processing priority and encryption strength are dynamically adjusted, and personalized processing and encryption strategies are formulated to ensure that the dynamic adjustment mechanism can ensure that the data weight can be adjusted in time with business changes and keep pace with business needs.

[0072] S5. Based on the formulation of corresponding processing and encryption strategies, preset rules for conflict decision-making on data processing priority and encryption priority are set.

[0073] Specifically, rules are set to resolve potential conflicts between data processing and encryption priorities, ensuring that critical operations can be performed efficiently even when resources are limited, balancing processing efficiency and security requirements, and avoiding system bottlenecks or security vulnerabilities caused by conflicts.

[0074] S6. According to the corresponding processing and encryption strategies and conflict decision preset rules, the system resources used for processing are allocated to each type of weighted data, and each type of data is analyzed, processed and encrypted to obtain archived data.

[0075] Specifically, appropriate system resources are allocated to each type of data based on the data's weight, processing priority, and encryption priority. The data is then analyzed and processed to extract valuable information. At the same time, the data is encrypted and protected according to the encryption strategy. Finally, the processed data is archived and stored.

[0076] S7. Develop access rights management policies and monitor data access based on archived data for use by internal applications, partners, or customers to ensure data security and user privacy.

[0077] Specifically, based on the characteristics and usage requirements of archived data, an access rights management strategy is formulated to clarify the access rights of different users or roles. At the same time, a data access monitoring mechanism is established to monitor data access in real time and provide data access services for internal applications, partners or customers.

[0078] Step S1 is specifically as follows:

[0079] S1.1. Determine the specific objectives and scope of data management, including the frequency, types, and sources of data collection;

[0080] S1.2. Design a data collection strategy based on the specific goals and scope of data management, including determining the timing of data collection, the selection of data sources, and the collection methods;

[0081] S1.3. Based on the data collection strategy, deploy big data collection tools and technologies to conduct dynamic data collection and establish data collection logs.

[0082] Step S2 is specifically as follows:

[0083] S2.1. Normalize the data, break it down into different tables, and use a relational database management system to manage the data.

[0084] S2.2. Before data is stored in the database, data cleaning work should be carried out, including removing duplicate records, correcting erroneous data, and processing missing values.

[0085] S2.3. During data processing, the data shall be checked and verified to ensure its accuracy and consistency.

[0086] S2.4. Use data matching and association technology to merge the pre-processed different source data to form a unified data set, and store the data set.

[0087] Step S3 is specifically as follows:

[0088] S3.1. Classify the data in the dataset according to its characteristics and usage to obtain various types of classified data. The data classification specifically includes user data, transaction data, and log data.

[0089] Specifically, a comprehensive review of the data set is conducted to identify the characteristic attributes of different data, such as data source, structure, content, etc. For example, user data may contain personal information such as name, age, and contact information; transaction data involves product number, transaction time, amount, etc.; log data records system activities, access timestamps, etc.

[0090] S3.2. Based on business needs and security levels, evaluate the importance and sensitivity of each type of classified data and obtain evaluation results.

[0091] Specifically, a scoring system or expert evaluation method is used to assess the importance of data based on its impact on business decision support, customer experience improvement, etc., and the sensitivity of the data is assessed based on laws, regulations, industry standards and personal privacy protection requirements. For example, personal identity information and health records are highly sensitive, and the final evaluation result is obtained by combining importance and sensitivity.

[0092] S3.3. Based on the evaluation results, establish specific rules and algorithms for data weight calculation to ensure that the final weight distribution is reasonable and reflects the actual situation.

[0093] Specifically, based on the evaluation results, the model of data weight calculation rules, such as the weighted average method and the hierarchical analysis method, assigns weights to the data, and the model needs to consider factors such as data volume, update frequency, and business impact. The model of data weight calculation rules is run through historical data or simulated data to test the rationality of the weight distribution. Adjustments and optimizations are made based on feedback to ensure that the weight distribution reflects the actual value of the data. The final data weight calculation rules and algorithms, specifically including calculation formulas, parameter settings, and update mechanisms, are determined.

[0094] Step S4 is specifically as follows:

[0095] S4.1. Adjust data weight calculation rules regularly or in real time based on business changes and actual needs.

[0096] Specifically, based on the current business development status, market trends and regulatory changes, ensure that data management strategies are synchronized with business needs, and use automated tools to monitor data flows and business indicators. When significant changes are detected, the evaluation process is automatically triggered. Based on the evaluation results and monitoring feedback, the data weight calculation rules are revised, including adjusting the weight allocation logic, adding or deleting considerations, etc.

[0097] S4.2. Based on the adjusted data weight calculation rules, define clear trigger conditions such as the addition of new data sources, and establish a dynamic adjustment mechanism through adaptive algorithms to assign new weights to various types of data after the triggering conditions in real time.

[0098] Specific and clear trigger conditions, such as the access of new data sources, data leakage incidents and regulatory updates, will initiate the recalculation of data weights. A dynamic adjustment mechanism will be established through adaptive algorithms. When the trigger conditions are met, the adaptive algorithms will automatically run, calculate the new weights of the data based on the latest data and rules, and instantly update the data processing and encryption strategies.

[0099] S4.3. Develop processing and encryption strategies based on the weights assigned to each type of data.

[0100] Specifically, based on the weights assigned to each type of data after the update, data processing and encryption strategies are designed for different categories of data. The processing strategy specifically covers the steps of data cleaning, integration, and analysis; while the encryption strategy determines the strength and method of data encryption. Ultimately, the strategy is deployed in the data management system to ensure that data processing and encryption operations are performed as required by the strategy.

[0101] Step S5 is specifically as follows:

[0102] S5.1. Analyze possible resource conflicts between processing priority and encryption priority.

[0103] Specifically, by simulating and analyzing historical data processing processes, we can identify which data processing tasks and encryption operations may conflict due to resource competition when resources are limited. For example, high-priority real-time transaction data processing and sensitive data encryption that also require high-performance resources are quantified. The resource requirements of conflicting tasks are also quantified, including computing resources (CPU, memory), storage resources, network bandwidth, etc., to understand their demand peaks and baselines in different time periods, analyze the impact of these conflicts on system performance, data security, and business continuity, and assess the potential risk level.

[0104] S5.2. Based on resource conflicts, define decision factors, including data importance, business impact, compliance requirements, and resource cost factors.

[0105] S5.3. Set conflict decision preset rules based on the defined decision elements, including priority rules, resource allocation rules, and emergency handling rules.

[0106] Specifically, according to the importance of data and business impact, set the priority of task execution to ensure that high-priority tasks are given priority; according to the priority and real-time resource status, automatically or manually adjust resource allocation to prioritize the supply of resources for critical tasks; formulate emergency response plans for data processing and encryption tasks when resources are extremely tight or system failures occur, such as data temporary storage, task downgrade processing, etc.

[0107] Step S6 is specifically as follows:

[0108] S6.1. Allocate computing and storage resources to each type of data based on its weight and processing priority.

[0109] Specifically, based on the data weight and processing priority, the demand for computing resources (CPU, memory) and storage resources for processing tasks of different types of data is analyzed, and according to the analysis results, appropriate computing nodes, storage space and network bandwidth are allocated to each type of data to ensure that high-priority and high-weight data receive priority resource supply. The resource allocation strategy is automatically adjusted through the resource management system of the data center to realize dynamic allocation of resources.

[0110] S6.2. Monitor resource usage to ensure that high-weight data receives sufficient resource support.

[0111] Specifically, resource monitoring tools are used to continuously track the usage of computing resources and storage resources, including CPU load, memory usage, storage space occupancy, etc., and performance monitoring thresholds are set for key data processing tasks. When resource usage approaches or reaches the threshold, an early warning is triggered. Based on the monitoring results, resource allocation is adjusted automatically or manually to ensure the smooth processing of high-weight data.

[0112] S6.3. When resources are insufficient or conflicts occur, real-time adjustments and allocations are made based on the preset conflict decision rules.

[0113] Specifically, when a resource conflict is detected, the previously set conflict decision preset rules are activated, such as resource priority adjustment, task downgrade or suspension. According to the preset rules, resources are reallocated automatically or manually to give priority to the resource requirements of high-priority data processing tasks.

[0114] Step S7 is specifically as follows:

[0115] S7.1. Role-based access control (RBAC) or attribute-based access control (ABAC) to set data access permissions for different users / roles.

[0116] S7.2. Implement data access logging, monitor abnormal access behavior, and conduct regular security audits.

[0117] S7.3. Provide internal users with data protection awareness and compliance training to improve overall data security management to ensure data security and user privacy rights.

[0118] Example 2

[0119] This embodiment provides a dynamic data management system, the management system including:

[0120] The data collection module is used to clarify the needs of dynamic data management and perform dynamic data collection based on big data.

[0121] Specifically, the data collection module is used to clarify the goals, scope and specific needs of data management, and then formulate data collection strategies based on these needs, including the frequency, data type, source, etc. of collection, and use big data collection tools or technologies to obtain data from various data sources in real time or in batches to ensure that the collected data is relevant, accurate and useful, providing a solid foundation for subsequent data processing and analysis.

[0122] The data integration module is used to pre-process data from different sources, merge the pre-processed data from different sources using data matching and association technology to form a unified data set, and store the data set.

[0123] Specifically, the data integration module is used to pre-process data from different sources. Pre-processing includes steps such as cleaning, conversion, and format unification to eliminate errors, redundancies, and inconsistencies in the data. Then, data matching and association technologies are used to merge data from different sources to form a complete and unified data set. Finally, an appropriate storage solution is selected to store the data set efficiently and securely to eliminate data differences and conflicts and improve data consistency and accuracy.

[0124] The weight setting module is used to classify the data in the data set and determine the weight according to its importance and sensitivity. It also formulates data weight calculation rules for each type of classified data.

[0125] Specifically, the weighting module is used to classify the data in the dataset according to the characteristics and usage of the data, then evaluate the importance and sensitivity of each type of data, determine the corresponding weight, and finally formulate weight calculation rules to provide a clear basis for weight assignment for different types of data, so as to more clearly understand the value and risks of various types of data, and provide an important basis for subsequent data processing and encryption strategy formulation.

[0126] The weight assignment module is used to establish a dynamic adjustment mechanism to assign weights to various types of data based on data weight calculation rules, and to formulate corresponding processing and encryption strategies.

[0127] Specifically, the weight assignment module is used to dynamically adjust data processing priority and encryption strength according to data weight calculation rules, and formulate personalized processing and encryption strategies to ensure that the dynamic adjustment mechanism can ensure that data weights can be adjusted in time with business changes and keep pace with business needs.

[0128] The decision module is used to set conflict decision preset rules for data processing priority and encryption priority based on the formulation of corresponding processing and encryption strategies.

[0129] Specifically, the decision module is used to set rules to resolve potential conflicts between data processing and encryption priorities, ensuring that key operations can be performed efficiently even when resources are limited, balancing processing efficiency and security requirements, and avoiding system bottlenecks or security vulnerabilities caused by conflicts.

[0130] The allocation processing module is used to allocate system resources for processing various types of weighted data according to the corresponding processing and encryption strategies and conflict decision preset rules, and analyze, process and encrypt various types of data to obtain archived data.

[0131] Specifically, the allocation processing module is used to allocate appropriate system resources to various types of data based on the data's weight, processing priority, and encryption priority. It then analyzes and processes the data to extract valuable information, encrypts and protects the data according to encryption policies, and finally archives and stores the processed data.

[0132] The access module is used to formulate access rights management policies and monitor data access based on archived data for use by internal applications, partners, or customers to ensure data security and user privacy.

[0133] Specifically, the access module is used to formulate access rights management strategies based on the characteristics and usage requirements of archived data, clarify the access rights of different users or roles, and establish a data access monitoring mechanism to monitor data access in real time and provide data access services for internal applications, partners or customers.

[0134] The weight setting module includes:

[0135] The classification module is used to classify the data in the dataset according to the characteristics and usage of the data, and obtain various types of classified data. The data classification specifically includes user data, transaction data and log data.

[0136] Specifically, the classification module is used to conduct a comprehensive review of the data set and identify the characteristic attributes of different data, such as data source, structure, content, etc. For example, user data may contain personal information such as name, age, and contact information; transaction data involves product number, transaction time, amount, etc.; log data records system activities, access timestamps, etc.

[0137] The evaluation module is used to evaluate the importance and sensitivity of various types of classified data based on business needs and security levels to obtain evaluation results.

[0138] Specifically, the evaluation module is used to use a scoring system or expert evaluation method to evaluate the importance of data based on the degree of impact of the data on business decision support, user experience improvement, etc., and to evaluate the sensitivity of the data based on laws, regulations, industry standards and personal privacy protection requirements. For example, personal identity information and health records are highly sensitive, and the final evaluation results are obtained by combining importance and sensitivity.

[0139] The rule setting module is used to establish specific rules and algorithms for data weight calculation rules based on the evaluation results to ensure that the final weight distribution is reasonable and reflects the actual situation.

[0140] Specifically, the rule setting module is used to assign weights to data based on the evaluation results and the model of data weight calculation rules, such as the weighted average method and the hierarchical analysis method. The model needs to consider factors such as data volume, update frequency, and business impact. The model of data weight calculation rules is run through historical data or simulated data to test the rationality of weight distribution. Adjustments and optimizations are made based on feedback to ensure that the weight distribution reflects the actual value of the data. The final data weight calculation rules and algorithms include calculation formulas, parameter settings, and update mechanisms.

[0141] The weighting module includes:

[0142] The adjustment module is used to adjust the data weight calculation rules regularly or in real time according to business changes and actual needs.

[0143] Specifically, the adjustment module is used to ensure that data management strategies are synchronized with business needs based on the current business development status, market trends and regulatory changes, and to use automated tools to monitor data flows and business indicators. When significant changes are detected, the evaluation process is automatically triggered. Based on the evaluation results and monitoring feedback, the data weight calculation rules are revised, including adjusting the weight allocation logic, adding or deleting considerations, etc.

[0144] The dynamic identification module is used to calculate data weights based on the adjusted rules, define clear trigger conditions such as the addition of new data sources, and establish a dynamic adjustment mechanism through adaptive algorithms to assign new weights to various types of data in real time after the trigger conditions.

[0145] Specifically, the dynamic identification module is used to identify trigger conditions, such as new data source access, data leakage incidents, and regulatory updates. It will initiate the recalculation of data weights and establish a dynamic adjustment mechanism through an adaptive algorithm. When the trigger conditions are met, the adaptive algorithm automatically runs to calculate the new weight of the data based on the latest data and rules, and instantly update the data processing and encryption strategies.

[0146] The policy formulation module is used to formulate processing policies and encryption policies based on the weights assigned to various types of data.

[0147] Specifically, the policy formulation module is used to design data processing and encryption strategies for different categories of data based on the weights assigned to each type of data after the update. The processing strategy specifically covers the steps of data cleaning, integration, and analysis; while the encryption strategy determines the strength and method of data encryption. Ultimately, the strategy is deployed to the data management system to ensure that data processing and encryption operations are performed according to the policy requirements.

[0148] The decision-making module includes:

[0149] An analysis module is used to analyze possible resource conflicts between processing priority and encryption priority.

[0150] Specifically, the analysis module is used to simulate and analyze historical data processing processes to identify which data processing tasks and encryption operations may conflict due to resource competition when resources are limited. For example, high-priority real-time transaction data processing and sensitive data encryption that also requires high-performance resources, and quantify the resource requirements of conflicting tasks, including computing resources (CPU, memory), storage resources, network bandwidth, etc., understand their demand peaks and baselines in different time periods, analyze the impact of these conflicts on system performance, data security, and business continuity, and evaluate the potential risk level.

[0151] Define the identification module, which is used to define decision factors based on resource conflicts, including data importance, business impact, compliance requirements and resource cost factors.

[0152] The setting module is used to set conflict decision preset rules based on the defined decision elements, including priority rules, resource allocation rules and emergency handling rules.

[0153] Specifically, the setting module is used to set the priority of task execution based on data importance and business impact, ensuring that high-priority tasks are given priority; automatically or manually adjust resource allocation based on priority and real-time resource status to prioritize the supply of resources for critical tasks; and formulate emergency response plans for data processing and encryption tasks when resources are extremely tight or system failures occur, such as data temporary storage, task downgrade processing, etc.

[0154] The allocation processing module includes:

[0155] The first-level scheduling module is used to allocate computing and storage resources to various types of data based on the data's weight and processing priority.

[0156] Specifically, the first-level scheduling module is used to analyze the demand for computing resources (CPU, memory) and storage resources for processing tasks of different types of data based on the data weight and processing priority, and allocate appropriate computing nodes, storage space and network bandwidth to each type of data based on the analysis results, to ensure that high-priority and high-weight data receive priority resource supply, and automatically adjust the resource allocation strategy through the data center's resource management system to achieve dynamic resource allocation.

[0157] The monitoring module is used to monitor the usage of computing resources and storage resources to ensure that high-weight data has sufficient resource support.

[0158] Specifically, the monitoring module is used to use resource monitoring tools to continuously track the usage of computing resources and storage resources, including CPU load, memory usage, storage space occupancy, etc., set performance monitoring thresholds for key data processing tasks, and trigger an early warning when resource usage approaches or reaches the threshold. Based on the monitoring results, resource allocation is adjusted automatically or manually to ensure the smooth processing of high-weight data.

[0159] The secondary scheduling module is used to make real-time adjustments and allocations based on the preset conflict decision rules when resources are insufficient or conflicts occur.

[0160] Specifically, the secondary scheduling module is used to activate previously set conflict decision preset rules when a resource conflict is detected, such as resource priority adjustment, task downgrade or suspension. According to the preset rules, resources are reallocated automatically or manually to give priority to the resource requirements of high-priority data processing tasks.

[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0163] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

Claims

1. A dynamic data management method, characterized in that: The management method comprises the following steps: S1. Clarify the needs of dynamic data management and collect dynamic data based on big data; S2. Preprocess the data from different sources, merge the preprocessed data from different sources using data matching and association technology to form a unified data set, and store the data set; S3. Classify the data in the data set and determine the weight according to its importance and sensitivity. Formulate data weight calculation rules for each type of classified data; S4. Based on the data weight calculation rules, a dynamic adjustment mechanism is established to assign weights to various types of data, and corresponding processing and encryption strategies are formulated; S5. Based on the formulation of corresponding processing and encryption strategies, set the conflict decision preset rules of data processing priority and encryption priority; S6. According to the corresponding processing and encryption strategies and conflict decision preset rules, the system resources used for processing are allocated to the various types of data after being assigned weights, and the various types of data are analyzed, processed and encrypted to obtain archived data; S7. Develop access rights management policies and monitor data access based on archived data for use by internal applications, partners, or customers to ensure data security and user privacy rights.

2. A dynamic data management method according to claim 1, characterized in that: The step S3 comprises the following steps: S3.

1. Classify the data in the data set according to the characteristics and uses of the data to obtain various types of classified data. The data classification specifically includes user data, transaction data and log data; S3.

2. Evaluate the importance and sensitivity of each type of data after classification based on business needs and security levels, and obtain evaluation results; S3.

3. Based on the evaluation results, establish specific rules and algorithms for data weight calculation to ensure that the final weight distribution is reasonable and reflects the actual situation.

3. A dynamic data management method according to claim 1, characterized in that: The step S4 comprises the following steps: S4.

1. Adjust the data weight calculation rules regularly or in real time according to business changes and actual needs; S4.

2. Based on the adjusted data weight calculation rules, define clear trigger conditions for the addition of new data sources, and establish a dynamic adjustment mechanism through adaptive algorithms to assign new weights to various types of data after the trigger conditions in real time; S4.

3. Develop processing and encryption strategies based on the weights assigned to each type of data.

4. A dynamic data management method according to claim 1, characterized in that: The step S5 comprises the following steps: S5.

1. Analyze the resource conflicts that may arise between processing priority and encryption priority; S5.

2. Define decision factors based on resource conflicts, including data importance, business impact, compliance requirements, and resource cost factors; S5.

3. Set conflict decision preset rules based on the defined decision elements, including priority rules, resource allocation rules and emergency handling rules.

5. A dynamic data management method according to claim 1, characterized in that: The step S6 comprises the following steps: S6.

1. Allocate computing resources and storage resources to each type of data based on the data weight and processing priority; S6.

2. Monitor resource usage to ensure that high-weight data has sufficient resource support; S6.

3. When resources are insufficient or conflicts occur, they are adjusted and allocated in real time according to the preset conflict decision rules.

6. A dynamic data management system, characterized in that: The management system comprises: Data collection module, used to clarify the needs of dynamic data management and perform dynamic data collection based on big data; The data integration module is used to pre-process data from different sources, merge the pre-processed data from different sources using data matching and association technology to form a unified data set, and store the data set; The weighting module is used to classify the data in the data set and determine the weight according to its importance and sensitivity. It also formulates data weight calculation rules for each type of classified data. The weight assignment module is used to establish a dynamic adjustment mechanism to assign weights to various types of data based on data weight calculation rules, and formulate corresponding processing and encryption strategies; A decision module, for setting conflict decision preset rules for data processing priority and encryption priority based on the formulation of corresponding processing and encryption strategies; The allocation processing module is used to allocate system resources for processing various types of data after weighting according to the corresponding processing and encryption strategies and conflict decision preset rules, and analyze, process and encrypt various types of data to obtain archived data; The access module is used to formulate access rights management policies and monitor data access based on archived data for use by internal applications, partners or customers to ensure data security and user privacy rights.

7. A dynamic data management system according to claim 6, characterized in that: The weight setting module includes: The classification module is used to classify the data in the data set according to the characteristics and uses of the data, and obtain various types of classified data. The data classification specifically includes user data, transaction data, and log data; The evaluation module is used to evaluate the importance and sensitivity of various types of classified data according to business needs and security levels, and obtain evaluation results; The rule setting module is used to establish specific rules and algorithms for data weight calculation rules based on the evaluation results to ensure that the final weight distribution is reasonable and reflects the actual situation.

8. A dynamic data management system according to claim 6, characterized in that: The weighting module comprises: The adjustment module is used to adjust the data weight calculation rules regularly or in real time according to business changes and actual needs; Dynamic identification module, which is used to define clear trigger conditions for the addition of new data sources based on the adjusted data weight calculation rules, and establish a dynamic adjustment mechanism through adaptive algorithms to assign new weights to various types of data after the trigger conditions in real time; The policy formulation module is used to formulate processing strategies and encryption strategies based on the weights assigned to various types of data.

9. A dynamic data management system according to claim 6, characterized in that: The decision module comprises: An analysis module for analyzing possible resource conflicts between processing priority and encryption priority; Define identification module, which is used to define decision factors according to resource conflicts, including data importance, business impact, compliance requirements and resource cost factors; The setting module is used to set conflict decision preset rules based on defined decision elements, including priority rules, resource allocation rules and emergency handling rules.

10. A dynamic data management system according to claim 6, characterized in that: The allocation processing module includes: The first-level scheduling module is used to allocate computing resources and storage resources to various types of data based on the weight and processing priority of the data; The monitoring module is used to monitor the usage of computing resources and storage resources to ensure that high-weight data has sufficient resource support; The secondary scheduling module is used to make real-time adjustments and allocations according to the preset conflict decision rules when resources are insufficient or conflicts occur.

Images