Management strategy method and system based on data life cycle

Abstract

The invention discloses a method and system based on a data life cycle management strategy, and the method comprises the steps: S1, a data creation stage, S2, a data storage stage, S3, a data processing stage, S4, a data transmission stage, S5, a data sharing stage, and S6, a data destruction stage. The data quality is kept stable for a long time, data creation specifications and strategies such as storage, processing and transmission can be dynamically adjusted according to service requirements, the data can better adapt to service changes, and automation and standardization are achieved in the data management process. Through dynamic adjustment of a storage strategy, overall planning of storage resources, optimization of transmission efficiency and other measures, the full life cycle of data is managed, a corresponding management strategy is formulated accordingly, and the data management efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data management, and in particular to a data lifecycle management strategy method and system. Background Art

[0002] With the rapid development of information technology, the amount of data has exploded. Enterprises and organizations face many challenges in data management. At present, common data management methods mainly focus on data storage and backup. Some enterprises use traditional database management systems to store data and prevent data loss through regular backup. In terms of data processing, some general data analysis tools are mainly used for simple analysis and processing. The main disadvantages of the existing technology are: data creation lacks pre-planning, and the existing technology often lacks the collection and organization of demand information in the data creation stage. This leads to a lack of basis for data creation and may not meet the needs of subsequent business. For example, in a sales system, if detailed requirements for sales data (sales area, product category, customer group, etc.) are not collected in advance, the sales data created may not accurately reflect the business situation, affecting subsequent data analysis and decision-making. Node management is not comprehensive. Traditional data management methods usually only focus on business data, while ignoring the detection of operational data and management indicators. This makes it impossible to fully understand the status of data at each node in the data life cycle, and it is difficult to ensure that the data meets expectations. When data problems occur, it is impossible to accurately trace the cause and it is difficult to optimize subsequent data processing. Lack of information tracing and prediction capabilities. Existing technologies lack effective information tracing and prediction methods in the data management process. When data problems occur, it is difficult to quickly locate the link and cause of the problem. At the same time, it is also impossible to predict the future development trend of the data, which is not conducive to taking measures in advance for optimization and adjustment. Summary of the invention

[0003] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a data lifecycle management strategy method and system.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A data lifecycle management strategy method includes the following steps:

[0006] S1: Data creation phase: First, understand the business problems and goals through the business understanding module, collect data through the acquisition module, organize the collected data through the business module and create data specifications, and then automatically analyze business needs through the auxiliary analysis module;

[0007] S2: Data storage stage: the data is encrypted by the data encryption module, and then the encrypted data is accessed through the access permission module to set access control, and the data is verified by the data verification module to prevent the data from being tampered with during storage;

[0008] S3: In the data processing stage, the distributed Apache Spark framework in the processing module is used to calculate the data. The data mining module uses the KNIME tool to mine and analyze the data, and the quality monitoring module is used to ensure the accuracy and completeness of the data processing results.

[0009] S4: Data transmission stage: the transmission encryption module uses the SSL / TLS protocol to encrypt and authenticate data to prevent data from being stolen and tampered with, and the network monitoring module uses Wireshark to monitor the data transmission process in real time. The real-time detection includes network failures, performance problems and security threats.

[0010] S5: Data sharing stage: Based on the sensitivity and usage requirements of the data, sharing rules and permissions are formulated through the intelligent policy module, and the blockchain mechanism is used to allow transparent sharing of data information in the enterprise network;

[0011] S6: Data destruction stage. Data destruction mainly includes physical destruction and logical destruction. Data destruction records must not only be archived, but also audited through the destruction audit module. The audit content includes whether the destruction process complies with the prescribed procedures and whether the destruction is thorough.

[0012] Preferably, the auxiliary analysis module also includes design optimization of data structure, definition of data fields, and data verification rules during automatic analysis, and the data structure design optimization is mainly optimized using a machine learning algorithm.

[0013] Preferably, the specific steps of creating the data specification are as follows:

[0014] a: Identify data types, and classify and identify the collected data according to common standards. The data types include but are not limited to description, code, value, date, and ID;

[0015] b: Business attribute definition: Based on the classification results, the business attributes of these data items are defined according to business standards;

[0016] c: Technical attribute definition: Based on the business attribute definition, clarify the business requirements for the technical attributes of data items, generate technical standard definitions, and generate common data standards;

[0017] d: Review and implementation: The generated common data standards shall be reviewed within the group, reviewed by experts and reviewed within the enterprise. After passing the review, the standards shall be implemented.

[0018] Preferably, the distributed Apache Spark framework can distribute large-scale data processing tasks to multiple computing nodes for execution, thereby improving the efficiency and speed of data processing.

[0019] Preferably, the present invention also provides a system based on the method of data lifecycle management strategy, the system includes a data creation module, a data storage module, a data processing module, a data transmission module, a data sharing module and a data destruction module, wherein the data creation module also includes a business understanding module, an acquisition module, a business module and an auxiliary analysis module, the business understanding module has an in-depth understanding of business problems and goals, and provides guidance for data collection and analysis, the data acquisition module obtains information through questionnaires, interviews, and meetings, and formulates data creation specifications through the business module, the auxiliary analysis module can automatically analyze business needs, the data storage module uses a hash algorithm through a data integrity check and mechanism to ensure the consistency and integrity of data during the storage process, the data processing module ensures the accuracy and integrity of the data processing results, the data transmission module encrypts and authenticates the data to prevent the data from being stolen and tampered with, the data sharing module determines which data can be shared and which data cannot be shared according to the sensitivity and usage requirements of the data, and the data destruction module mainly ensures data security and non-leakage.

[0020] Preferably, the business module includes an identification module, a classification module and an audit module. The identification module classifies and identifies different types of data according to the type, and defines and classifies different types of data through the classification module. The audit module mainly audits the generated general data standards, and executes them according to the standards after the audit is passed.

[0021] Preferably, the data processing module includes a processing module, a data mining module and a quality monitoring module. The processing module mainly adopts the Apache Spark framework to perform efficient data processing and analysis. The data mining module adopts KNIME to perform in-depth data analysis and mining. The quality monitoring module also includes a screening module and an allocation module. The screening module performs screening by setting parameters such as data quality, data field, and data timeliness. After screening, accurate allocation is achieved through the allocation module.

[0022] Preferably, the data transmission module includes a transmission encryption module and a network monitoring module, and the data sharing module includes an intelligent strategy module and a blockchain module. The data transmission process can be monitored through encrypted transmission and network monitoring, and potential network problems can be discovered and resolved in a timely manner. The data sharing module can anonymize shared data to protect the privacy of the data.

[0023] Preferably, the data destruction module further includes a destruction module and a destruction review module, and the destruction review module reviews whether the destruction process complies with prescribed procedures and whether the destruction is thorough.

[0024] The beneficial effects of the present invention are:

[0025] 1. Improved data quality. Through continuous demand information collection and integration, strict node management and standardized review processes, data quality remains stable in the long term. In addition, since data creation specifications and storage, processing, transmission and other strategies can be dynamically adjusted according to business needs, data can better adapt to business changes.

[0026] 2. Enhanced the traceability and controllability of data management. Through a comprehensive node management framework, detailed operation records are kept at every stage of the data life cycle, including business data, operation data, and management indicator records. These records provide a detailed basis for tracing and solving data problems. Through dynamic shared permission management and strict access control, the controllability of data management is enhanced. Access rights can be flexibly allocated according to the roles and needs of different users to ensure that only authorized personnel can access and use data.

[0027] 3. Improved data management efficiency, many links in the data management process have been automated and standardized. Through dynamic adjustment of storage strategies, overall planning of storage resources, optimization of transmission efficiency and other measures, storage resources and transmission resources have been optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the method of the present invention;

[0029] Figure 2 A flow chart for creating data specifications for the present invention;

[0030] Figure 3 It is a structural flow chart of the system of the present invention;

[0031] Figure 4 A system diagram for the data creation module of the present invention;

[0032] Figure 5 A system diagram of a data storage module of the present invention;

[0033] Figure 6 is a system diagram of the data processing module of the present invention;

[0034] Figure 7 A system diagram of a data transmission module of the present invention;

[0035] Figure 8 It is a system diagram of the data sharing module of the present invention;

[0036] Fig. 9 This is a system diagram of the data destruction module of the present invention; DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0038] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention. In the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0039] Embodiment 1:

[0040] See also Figure 1 , a method based on data lifecycle management strategy, comprising the following steps:

[0041] S1: Data creation phase: First, understand business problems and goals through the business understanding module, mainly through the CRISP-DM model, and collect data through the acquisition module. The collected data is sorted through the business module and data specifications are created, including the design of data structure, definition of data fields, data validation rules, etc. Then the business needs are automatically analyzed through the auxiliary analysis module;

[0042] S2: Data storage stage, using high-performance database systems, such as data management systems based on non-volatile memory (NVM), can improve data storage efficiency and speed. Then implement data encryption and access control measures, encrypt data through the data encryption module, and then set access control for the encrypted data through the access permission module, and verify the data through the data verification module to prevent data from being tampered with during storage;

[0043] S3: In the data processing stage, the distributed Apache Spark framework in the processing module is used to calculate the data. The data mining module uses the KNIME tool to mine and analyze the data, and the quality monitoring module is used to ensure the accuracy and completeness of the data processing results.

[0044] S4: Data transmission stage: the transmission encryption module uses the SSL / TLS protocol to encrypt and authenticate data to prevent data from being stolen and tampered with, and the network monitoring module uses Wireshark to monitor the data transmission process in real time. The real-time detection includes network failures, performance problems and security threats.

[0045] S5: Data sharing stage: Based on the sensitivity and usage requirements of the data, sharing rules and permissions are formulated through the intelligent policy module, and the blockchain mechanism is used to allow transparent sharing of data information in the enterprise network;

[0046] S6: Data destruction stage. Data destruction mainly includes physical destruction and logical destruction. Data destruction records must not only be archived, but also audited through the destruction audit module. The audit content includes whether the destruction process complies with the prescribed procedures and whether the destruction is thorough.

[0047] The auxiliary analysis module also includes the design optimization of data structure, definition of data fields, and data verification rules during automatic analysis, and the data structure design optimization is mainly optimized using machine learning algorithms.

[0048] participate Figure 2 , the specific steps of creating the data specification are as follows:

[0049] a: Identify data types, and classify and identify the collected data according to common standards. The data types include but are not limited to description, code, value, date, and ID;

[0050] b: Business attribute definition: Based on the classification results, these data items are classified according to business standards to define business attributes; for example, software business corresponds to software data, and traffic business corresponds to traffic data

[0051] c: Technical attribute definition: Based on the business attribute definition, clarify the business requirements for the technical attributes of data items, generate technical standard definitions, and generate common data standards; such as the format and length of data items;

[0052] d: Review and implementation: The generated common data standards shall be reviewed within the group, reviewed by experts and reviewed within the enterprise. After passing the review, the standards shall be implemented.

[0053] The distributed Apache Spark framework can distribute large-scale data processing tasks to multiple computing nodes for execution, thereby improving the efficiency and speed of data processing.

[0054] participate Figure 3The present invention also provides a system based on the method of data lifecycle management strategy, the system includes a data creation module, a data storage module, a data processing module, a data transmission module, a data sharing module and a data destruction module, wherein the data creation module also includes a business understanding module, a collection module, a business module and an auxiliary analysis module, the business understanding module has an in-depth understanding of business problems and goals, and provides guidance for data collection and analysis, the data collection module obtains information through questionnaires, interviews, and meetings, and formulates data creation specifications through the business module, the auxiliary analysis module can automatically analyze business needs, the data storage module uses a hash algorithm through a data integrity check and mechanism to ensure the consistency and integrity of data during the storage process, the data processing module ensures the accuracy and integrity of the data processing results, the data transmission module encrypts and authenticates the data to prevent the data from being stolen and tampered with, the data sharing module determines which data can be shared and which data cannot be shared according to the sensitivity and usage requirements of the data, and the data destruction module mainly ensures data security and non-leakage.

[0055] See also Figure 4 The business module includes an identification module, a classification module and an audit module. The identification module classifies and identifies different types of data according to the type, and defines and classifies different types of data through the classification module. The audit module mainly audits the generated general data standards, and executes them according to the standards after the audit is passed.

[0056] See also Figure 5 , the data storage module includes a data encryption module, an access permission module and a data verification module. The use of a high-performance database system, such as a data management system based on non-volatile memory (NVM), can improve data storage efficiency and speed. In implementing data encryption and access control measures, data encryption can prevent data from being accessed and stolen without authorization, and access control measures can limit user access rights to data. Through the data integrity check and mechanism, a hash algorithm is used to ensure the consistency and integrity of data during the storage process. The hash algorithm can generate a unique identifier for the data, which is used to verify the integrity of the data, in order to prevent the data from being tampered with during the storage process.

[0057] See also Figure 6 The data processing module includes a processing module, a data mining module and a quality monitoring module. The processing module mainly adopts the Apache Spark framework for efficient data processing and analysis. The data mining module adopts KNIME for in-depth data analysis and mining. The quality monitoring module also includes a screening module and an allocation module. The screening module performs screening by setting parameters such as data quality, data field, and data timeliness. After screening, accurate allocation is achieved through the allocation module.

[0058] See also Figure 7-8 The data transmission module includes a transmission encryption module and a network monitoring module, and the data sharing module includes an intelligent strategy module and a blockchain module. The data transmission process can be monitored through encrypted transmission and network monitoring, and potential network problems can be discovered and resolved in a timely manner. The data sharing module can anonymize the shared data to protect the privacy of the data.

[0059] See also Fig. 9 The data destruction module also includes a destruction module and a destruction review module. The destruction review module reviews whether the destruction process complies with the prescribed procedures and whether the destruction is thorough.

[0060] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method based on a data lifecycle management strategy, characterized in that: The steps include: S1: Data creation phase: First, understand the business problems and goals through the business understanding module, collect data through the acquisition module, organize the collected data through the business module and create data specifications, and then automatically analyze business needs through the auxiliary analysis module; S2: Data storage stage: the data is encrypted by the data encryption module, and then the encrypted data is accessed through the access permission module to set access control, and the data is verified by the data verification module to prevent the data from being tampered with during storage; S3: In the data processing stage, the distributed Apache Spark framework in the processing module is used to calculate the data. The data mining module uses the KNIME tool to mine and analyze the data, and the quality monitoring module is used to ensure the accuracy and completeness of the data processing results. S4: Data transmission stage: the transmission encryption module uses the SSL / TLS protocol to encrypt and authenticate data to prevent data from being stolen and tampered with, and the network monitoring module uses Wireshark to monitor the data transmission process in real time. The real-time detection includes network failures, performance problems and security threats. S5: Data sharing stage: Based on the sensitivity and usage requirements of the data, sharing rules and permissions are formulated through the intelligent policy module, and the blockchain mechanism is used to allow transparent sharing of data information in the enterprise network; S6: Data destruction stage. Data destruction mainly includes physical destruction and logical destruction. Data destruction records must not only be archived, but also audited through the destruction audit module. The audit content includes whether the destruction process complies with the prescribed procedures and whether the destruction is thorough.

2. The method according to claim 1, characterized in that: The auxiliary analysis module also includes the design optimization of data structure, definition of data fields, and data verification rules during automatic analysis, and the data structure design optimization is mainly optimized using machine learning algorithms.

3. The method according to claim 1, characterized in that: The specific steps of creating the data specification are as follows: a: Identify data types, and classify and identify the collected data according to common standards. The data types include but are not limited to description, code, value, date, and ID; b: Business attribute definition: Based on the classification results, the business attributes of these data items are defined according to business standards; c: Technical attribute definition: Based on the business attribute definition, clarify the business requirements for the technical attributes of data items, generate technical standard definitions, and generate common data standards; d: Review and implementation: The generated common data standards shall be reviewed within the group, reviewed by experts and reviewed within the enterprise. After passing the review, the standards shall be implemented.

4. The method based on data lifecycle management strategy according to claim 1, characterized in that: The distributed Apache Spark framework can distribute large-scale data processing tasks to multiple computing nodes for execution, thereby improving the efficiency and speed of data processing.

5. A system based on data lifecycle management strategy, characterized in that: According to the method based on data lifecycle management strategy according to any one of claims 1 to 4, the system includes a data creation module, a data storage module, a data processing module, a data transmission module, a data sharing module and a data destruction module, wherein the data creation module also includes a business understanding module, an acquisition module, a business module and an auxiliary analysis module, the business understanding module has an in-depth understanding of business problems and goals, and provides guidance for data collection and analysis, the data acquisition module obtains information through questionnaires, interviews, and meetings, and formulates data creation specifications through the business module, the auxiliary analysis module can automatically analyze business needs, the data storage module uses a hash algorithm through a data integrity check and mechanism to ensure the consistency and integrity of data during the storage process, the data processing module ensures the accuracy and integrity of the data processing results, the data transmission module encrypts and authenticates the data to prevent the data from being stolen and tampered with, the data sharing module determines which data can be shared and which data cannot be shared according to the sensitivity and usage requirements of the data, and the data destruction module mainly ensures data security and non-leakage.

6. The system based on data lifecycle management strategy according to claim 5, characterized in that: The business module includes an identification module, a classification module and an audit module. The identification module classifies and identifies different types of data according to the types, and defines and classifies different types of data through the classification module. The audit module mainly audits the generated general data standards, and executes them according to the standards after the audit is passed.

7. The system based on data lifecycle management strategy according to claim 5, characterized in that: The data processing module includes a processing module, a data mining module and a quality monitoring module. The processing module mainly adopts the Apache Spark framework for efficient data processing and analysis. The data mining module adopts KNIME for in-depth data analysis and mining. The quality monitoring module also includes a screening module and an allocation module. The screening module performs screening by setting parameters such as data quality, data field, and data timeliness. After screening, accurate allocation is achieved through the allocation module.

8. The system based on data lifecycle management strategy according to claim 5, characterized in that: The data transmission module includes a transmission encryption module and a network monitoring module, and the data sharing module includes an intelligent strategy module and a blockchain module. The data transmission process can be monitored through encrypted transmission and network monitoring, and potential network problems can be discovered and resolved in a timely manner. The data sharing module can anonymize shared data to protect the privacy of the data.

9. The system based on data lifecycle management strategy according to claim 5, characterized in that: The data destruction module also includes a destruction module and a destruction review module. The destruction review module reviews whether the destruction process complies with the prescribed procedures and whether the destruction is thorough.

Images