Data asset sorting method

Abstract

The invention relates to the technical field of data management, in particular to a data asset sorting method, which comprises the following steps: preliminarily classifying all data assets in an enterprise, performing quality evaluation on the preliminarily classified data assets, generating a quality evaluation report, and performing data redundancy detection and optimization based on the quality evaluation result. Cleaning redundant data by adopting a deduplication algorithm and an optimization algorithm, carrying out data correlation analysis, generating a data correlation graph through a graph model and a statistical model, and determining the relationship between data sets; and finally, according to a correlation analysis result, integrating the data sets with high correlation into a unified data set by adopting a data fusion algorithm, and reducing a storage space through a data compression technology. According to the method, the accuracy of data classification and management can be improved, the data redundancy is reduced, the data storage and access efficiency is optimized, and the data security and privacy protection are enhanced through security evaluation and protection measures.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a method for combing data assets. Background Art

[0002] With the development of information technology, enterprises have accumulated a large amount of digital data assets in their daily operations. These data assets are of various types, including structured data (such as database records), semi-structured data (such as XML, JSON files) and unstructured data (such as text, images, videos). These data come from a wide range of sources, which may come from internal business systems, customer feedback, market research or external suppliers. How to effectively manage and utilize these data assets has become an important factor in improving the competitiveness of enterprises.

[0003] Existing technical means often face the following key problems in the data management process: First, data classification usually relies on static rules, lacks dynamic adjustment capabilities, and cannot flexibly respond to changes in corporate business; second, data quality assessment is often limited to a single dimension, lacks a comprehensive assessment of data integrity, accuracy, consistency and timeliness, resulting in potential risks in the subsequent use of data; finally, data redundancy, conflict and invalid data are common in enterprises, but existing redundancy detection and optimization methods are inefficient, resulting in increased data storage costs and reduced processing efficiency. Summary of the invention

[0004] Based on the above objectives, the present invention provides a method for combing data assets.

[0005] A method for combing data assets includes the following steps:

[0006] S1. Preliminary classification of data assets: Preliminary classification of all data assets within the enterprise. The classification criteria include data type, data source, data structure and data storage method. The preliminary classification adopts a rule-based classification algorithm to generate a classification model based on the characteristics and relevance of data assets, and dynamically adjusts and optimizes it according to business needs to ensure that the classification results are highly consistent with the business needs of the enterprise;

[0007] S2. Data asset quality assessment: Conduct quality assessment on the data assets after preliminary classification and generate a quality assessment report. The assessment indicators include data integrity, data accuracy, data consistency and data timeliness;

[0008] S3. Data redundancy detection and optimization: Based on the results of data asset quality assessment, data redundancy detection is performed. The detection content includes data duplication, invalidity and conflict. Data deduplication algorithm and optimization algorithm are used to clean and optimize redundant data to ensure the streamlining and efficiency of data assets and reduce unnecessary data storage occupation.

[0009] S4. Data correlation analysis: After data redundancy detection and optimization, data assets are analyzed for correlation. The analysis methods include correlation analysis based on graph models and correlation analysis based on statistical models. Data correlation graphs are generated to clarify the relationships between different data sets, providing a basis for data integration and further utilization.

[0010] S5. Data asset integration and classified storage: Based on the results of data correlation analysis, data fusion algorithms are used to integrate highly correlated data sets into a unified data set, and data compression technology is used to reduce data storage space. The integrated data assets are labeled according to business needs and classified and stored according to optimized classification standards to ensure efficient management and utilization of data assets;

[0011] S6. Data asset visualization: Visualize the classified and stored data assets and generate multi-dimensional display diagrams, including data distribution diagrams and correlation diagrams, to help enterprise managers fully understand and monitor the status of data assets.

[0012] Optionally, the preliminary classification of data assets in step S1 specifically includes:

[0013] S11, data feature extraction: extract multi-dimensional features from various data assets within the enterprise. The multi-dimensional features include data type, data source, data structure and data storage method;

[0014] S12, classification rule formulation: establishing classification rules based on the extracted features;

[0015] S13, classification model generation: preliminarily classify the data according to the established classification rules, generate a classification data set, apply the support vector machine algorithm to build a classification model, use the classification data set as training data to train the classification model, use the cross-validation technology to evaluate the model performance, adjust the model parameters according to the validation results, and improve the classification accuracy;

[0016] S14, classification result output: classify all data assets according to the classification model and generate a classification report. The report content includes the classification results and classification basis.

[0017] Optionally, the data asset quality assessment in step S2 specifically includes:

[0018] S21: Data integrity assessment: Calculate the integrity of the data set, calculated as:

[0019]

[0020] Where I is the data integrity percentage, D c is the number of complete data items, Dt is the total number of data items;

[0021] S22: Data accuracy assessment: Calculate the accuracy of the data by comparing it with the reference data set, which is calculated as:

[0022]

[0023] Among them, A is the data accuracy percentage, D a is the exact number of data items, D t is the total number of data items;

[0024] S23: Data consistency assessment: Evaluate the consistency of data between different systems, calculated as:

[0025]

[0026] Among them, C is the data consistency percentage, D s is the number of consistent data items, D t is the total number of data items;

[0027] S24: Data timeliness evaluation: Calculate data timeliness based on timestamps, calculated as:

[0028]

[0029] Among them, T is the data timeliness percentage, T n is the current time, T d The latest update time of the data, T o The time when the data was created;

[0030] S25: Generation of quality assessment report: Integrate various assessment indicators to generate a data asset quality assessment report to provide a basis for subsequent processing.

[0031] Optionally, the data redundancy detection and optimization in step S3 specifically includes:

[0032] S31: Data duplication detection: Use hash algorithm to uniquely encode data items and detect the duplication of data items by generating hash values;

[0033] S32: Data invalidity detection: Identify invalid or low-value data items based on their frequency of use and relevance in business processes;

[0034] S33: Data conflict detection: After checking for data duplication and invalidity, check whether there are logical or numerical conflicts between data items;

[0035] S34: Data optimization: Based on the redundancy detection results, use deduplication and optimization algorithms to clean the data set, delete redundant data, merge or correct conflicting data, and finally generate an optimized data set.

[0036] Optionally, the data correlation analysis in step S4 specifically includes:

[0037] S41: Graph model construction: construct a graph model G(V,E) representing the relationship between data sets, where V represents data nodes and E represents the relationship edge between nodes;

[0038] S42: Association calculation: Based on the graph model, calculate the association strength between nodes;

[0039] S43: Statistical model analysis: Use statistical methods to perform correlation analysis on the data set, and the statistical method uses the Pearson correlation coefficient;

[0040] S44: Data association graph generation: Generate a data association graph based on the results of association calculation and statistical analysis.

[0041] Optionally, the data integration and classified storage in step S5 specifically includes:

[0042] S51: Data integration: Based on the results of correlation analysis, the highly correlated data sets are integrated. The principal component analysis method is used in the data integration process to reduce redundant dimensions.

[0043] S52: Data compression: After the data integration is completed, the integrated data is compressed using a lossless compression algorithm;

[0044] S53: labeling processing: labeling the compressed data. Labeling is to assign one or more labels to the data according to the attributes of the data. The attributes of the data include source, type and application scenario.

[0045] S54: Classification storage: Classify and store labeled data assets according to the optimized classification standards.

[0046] Optionally, the data asset visualization in step S6 specifically includes:

[0047] S61: Data distribution graph generation: Generate data distribution graphs of different dimensions based on the classified and stored data assets;

[0048] S62: Data association graph generation: Based on the association analysis results, a data association graph is generated. The association graph displays the association and relationship strength between data sets in the form of nodes and edges.

[0049] S63: Application of data visualization tools: Use data visualization tools or customized visualization platforms within the enterprise to display the generated data distribution diagrams and correlation diagrams.

[0050] Optionally, the optimization process of data asset classification storage in step S5 further includes:

[0051] S55: Distributed storage strategy formulation: Develop data distribution strategies based on the scale, data type, access frequency, and business needs of data assets;

[0052] S56: Data copy management: Generate multiple data copies to improve data accessibility and fault tolerance;

[0053] S57: Storage node optimization: Optimize and adjust storage nodes based on data access frequency and timeliness;

[0054] S58: Data access optimization: After data classification and storage are completed, a data access optimization strategy is formulated to improve data retrieval efficiency.

[0055] Optionally, the data asset visualization in step S6 further includes:

[0056] S64, Security assessment of visualization results: After generating visualization results, a comprehensive security assessment is conducted to ensure that sensitive data will not be leaked during the visualization process. Sensitive data includes employee identity information, financial data, corporate contract information, and corporate customer information.

[0057] S65, Data encryption display: For data with high sensitivity (such as financial data), dynamic encryption technology is used in visual display;

[0058] S66, Security Logging: During data visualization, all access and operations involving sensitive data are recorded in real time;

[0059] S67, Security report generation: Generate security reports regularly based on security assessments and log records.

[0060] Beneficial effects of the present invention:

[0061] The present invention ensures that enterprises can dynamically and accurately classify and manage their data assets through systematic data feature extraction, rule formulation, classification model generation and optimization. The comprehensive evaluation process of data integrity, accuracy, consistency and timeliness, combined with data redundancy detection and optimization, not only reduces redundancy and conflict in data, but also improves data utilization and processing efficiency, can greatly reduce human errors in data management, and improve the accuracy and efficiency of enterprises in data analysis and decision-making.

[0062] The present invention can effectively manage and store large-scale data assets through the formulation of distributed storage solutions, data copy management, storage node optimization and data access optimization. Enterprises can find the best balance between performance and cost, and ensure that high-frequency data can be quickly accessed by reasonably allocating and optimizing storage resources, while storing low-frequency data in more cost-effective storage media. This not only improves the storage and access efficiency of data, but also greatly reduces unnecessary storage costs and system loads.

[0063] The present invention, by introducing comprehensive security assessment, data encryption display and security log recording in the process of data visualization display, ensures that sensitive data will not be accessed or leaked by unauthorized users during the display and transmission process. The generation and real-time monitoring of security reports further enhance the enterprise's response capabilities to potential security threats, reduce the risk of data leakage and security accidents, and not only help enterprises meet compliance requirements, but also protect the privacy and security of individuals and enterprises, and enhance the security and reliability of data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0066] Figure 2 Schematic diagram of the process flow of step S5 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0068] It should be noted that the references to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0069] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0070] like Figure 1-Figure 2 As shown, a method for combing data assets includes the following steps:

[0071] S1. Preliminary classification of data assets: Preliminary classification of all data assets within the enterprise. The classification criteria include data type, data source, data structure and data storage method. The preliminary classification adopts a rule-based classification algorithm to generate a classification model based on the characteristics and relevance of data assets, and dynamically adjusts and optimizes it according to business needs to ensure that the classification results are highly consistent with the business needs of the enterprise;

[0072] S2. Data asset quality assessment: Conduct quality assessment on the data assets after preliminary classification and generate a quality assessment report. The assessment indicators include data integrity, data accuracy, data consistency and data timeliness;

[0073] S3. Data redundancy detection and optimization: Based on the results of data asset quality assessment, data redundancy detection is performed. The detection content includes data duplication, invalidity and conflict. Data deduplication algorithm and optimization algorithm are used to clean and optimize redundant data to ensure the streamlining and efficiency of data assets and reduce unnecessary data storage occupation.

[0074] S4. Data correlation analysis: After data redundancy detection and optimization, data assets are analyzed for correlation. The analysis methods include correlation analysis based on graph models and correlation analysis based on statistical models. Data correlation graphs are generated to clarify the relationships between different data sets, providing a basis for data integration and further utilization.

[0075] S5. Data asset integration and classified storage: Based on the results of data correlation analysis, data fusion algorithms are used to integrate highly correlated data sets into a unified data set, and data compression technology is used to reduce data storage space. The integrated data assets are labeled according to business needs and classified and stored according to optimized classification standards to ensure efficient management and utilization of data assets;

[0076] S6. Data asset visualization: Visualize the classified and stored data assets and generate multi-dimensional display diagrams, including data distribution diagrams and correlation diagrams, to help enterprise managers fully understand and monitor the status of data assets.

[0077] The preliminary classification of data assets in step S1 includes:

[0078] S11, data feature extraction: extract multi-dimensional features from various data assets within the enterprise. The multi-dimensional features include data type, data source, data structure and data storage method;

[0079] Data types include structured data, semi-structured data, and unstructured data. Data sources include internal systems, external suppliers, and customer feedback. Data structures include flat tables, tree structures, and chart structures. Data storage methods include databases, data warehouses, and cloud storage.

[0080] S12, classification rule formulation: According to the extracted features, classification rules are established. The establishment of rules is based on the business needs of the enterprise and the existing data management specifications, such as:

[0081] Data type rules: IF data type = "structured" THEN classify as "structured data";

[0082] Data source rule: IF data source = "internal system" THEN the classification is "internal data";

[0083] Data structure rules: IF data structure = "hierarchical structure" THEN classify as "hierarchical structure data";

[0084] Data storage method rule: IF data storage method = "cloud storage" THEN classify as "cloud data";

[0085] Rule priority: Set a priority for each classification rule to ensure that when the same data item meets multiple rules, the most appropriate classification is applied, for example, data type is prioritized, followed by data source, and so on;

[0086] S13, classification model generation: preliminarily classify the data according to the established classification rules, generate a classification data set, apply the support vector machine algorithm to build a classification model, use the classification data set as training data to train the classification model, use the cross-validation technology to evaluate the model performance, adjust the model parameters according to the validation results, and improve the classification accuracy;

[0087] S14, classification result output: classify all data assets according to the classification model and generate a classification report, which includes the classification results and classification basis;

[0088] The preliminary classification of data assets is carried out through a systematic approach to ensure the accuracy and flexibility of the classification. The rule-based classification algorithm enables the classification model to adapt to the dynamic changes in the business needs of the enterprise, thereby improving the efficiency and accuracy of data management.

[0089] The data asset quality assessment in step S2 specifically includes:

[0090] S21: Data integrity assessment: Calculate the integrity of the data set, calculated as:

[0091]

[0092] Where I is the data integrity percentage, D c is the number of complete data items, D t is the total number of data items;

[0093] S22: Data accuracy assessment: Calculate the accuracy of the data by comparing it with the reference data set, which is calculated as:

[0094]

[0095] Among them, A is the data accuracy percentage, D a is the exact number of data items, D t is the total number of data items;

[0096] S23: Data consistency assessment: Evaluate the consistency of data between different systems, calculated as:

[0097]

[0098] Among them, C is the data consistency percentage, D s is the number of consistent data items, D t is the total number of data items;

[0099] S24: Data timeliness evaluation: Calculate data timeliness based on timestamps, calculated as:

[0100]

[0101] Among them, T is the data timeliness percentage, T n is the current time, T d The latest update time of the data, T o The time when the data was created;

[0102] S25: Quality assessment report generation: Integrate various assessment indicators to generate a data asset quality assessment report to provide a basis for subsequent processing;

[0103] By evaluating the integrity, accuracy, consistency and timeliness of data through detailed indicators, potential problems in the data can be effectively discovered and resolved, providing a solid foundation for subsequent data optimization and processing, and ensuring the high quality and reliability of data assets.

[0104] Data redundancy detection and optimization in step S3 specifically include:

[0105] S31: Data duplication detection: Use the hash algorithm to uniquely encode the data item, and detect the duplication of the data item by generating a hash value. The hash value generation formula is expressed as:

[0106] H(x) = hash(x);

[0107] Where H(x) is the hash value of data item x, and hash is a hash function used to generate a unique hash value. By comparing hash values, duplicate data items can be quickly detected and deleted to avoid data redundancy.

[0108] S32: Data invalidity detection: Identify invalid or low-value data items based on the frequency of use of data items and their relevance in the business process. The calculation formula for data usage frequency is:

[0109]

[0110] Among them, F(x) is the usage frequency of data item x, U(x) is the actual number of times the data item is used, and U t The total number of times it is used. For data items that are less frequently used and no longer have business value, consider deleting or archiving them.

[0111] S33: Data conflict detection: After checking for data duplication and invalidity, check whether there are logical or numerical conflicts between data items. The conflict detection formula is:

[0112] C(x1,x2)=IF(x1≠x2) THEN conflict;

[0113] Wherein, C(x1, x2) is the conflict detection result between data items x1 and x2;

[0114] S34: Data optimization: Based on the redundancy detection results, use deduplication algorithms and optimization algorithms to clean up the data set, delete redundant data, merge or correct conflicting data, and finally generate an optimized data set;

[0115] The deduplication algorithm is used to detect and delete duplicate records or data items in the data set. Based on the hash algorithm in S31, a unique hash value is generated for each data item, each hash value is stored in a set, and compared with the existing hash values ​​in the set. If the hash value already exists, the data item is determined to be a duplicate and deleted. Otherwise, the hash value is added to the set.

[0116] The optimization algorithm is used to handle conflicts and invalidity issues between data items. Based on the conflict detection in S33, for conflicting data items (such as inconsistent values ​​of different fields in the same record), the most credible value is selected as the final value by analyzing the source, credibility, and timestamp of the data items.

[0117] A data item refers to an independent unit or record in a data set. For structured data, a data item can be a row of records or a field value in a database table. For unstructured data, a data item can be a paragraph of text in a text file, a segment of audio content in an audio file, etc. A data item is the most basic unit of analysis and processing. In the process of data redundancy detection and optimization, all operations are performed based on data items.

[0118] Through deduplication and optimization algorithms, redundant data and conflicting data in the data set can be effectively cleaned and integrated. The deduplication algorithm effectively reduces the occupancy of duplicate data, and the optimization algorithm improves the overall quality and consistency of the data set by correcting conflicts and integrating data, ensuring the efficiency and reliability of the data set, and providing a cleaner and more accurate basic data set for enterprise data analysis and decision-making.

[0119] The data correlation analysis in step S4 specifically includes:

[0120] S41: Graph model construction: construct a graph model G(V,E) representing the relationship between data sets, where V represents data nodes, E represents the relationship edges between nodes, data nodes represent data sets or data items, and relationship edges represent the association relationships between data sets (such as dependency relationships, shared data items, etc.);

[0121] S42: Association calculation: Based on the graph model, calculate the association strength between nodes, the association strength W between nodes i and j ij , calculated as:

[0122]

[0123] Among them, Wij is the association strength between nodes i and j, w ik and w kj is the weight, which indicates the strength of the relationship between nodes, d ij is the distance between nodes i and j or the length of the relationship chain. The correlation calculation helps to identify the close relationship between data sets, which is convenient for subsequent integration;

[0124] S43: Statistical model analysis: Use statistical methods to perform correlation analysis on the data set. The statistical method uses the Pearson correlation coefficient r, which is calculated as:

[0125]

[0126] Among them, r is the Pearson correlation coefficient, x i and i are the i-th data points of the two data sets respectively, and are the means of the two data sets, respectively. This analysis method helps to quantify the linear relationship between the data sets;

[0127] S44: Data association graph generation: Based on the results of association calculation and statistical analysis, a data association graph is generated. The graph displays the association and correlation between data sets in a visual form, providing an intuitive reference for enterprise managers and supporting data integration and decision-making;

[0128] Through data correlation analysis, the relationship between data sets can be accurately identified and quantified, which not only helps enterprises understand the interdependence of data, but also provides a scientific basis for data integration and promotes the further utilization and optimization of data assets.

[0129] The data integration and classified storage in step S5 specifically include:

[0130] S51: Data integration: Based on the results of correlation analysis, the highly correlated data sets are integrated. The principal component analysis method is used in the data integration process to reduce redundant dimensions and ensure the uniformity and consistency of the data set. The mathematical expression of the principal component analysis method is:

[0131] Z = XW;

[0132] Among them, Z is the data matrix after dimension reduction, X is the original data matrix, and W is the transformation matrix. The principal component analysis method is used to reduce the data dimension and improve the efficiency and accuracy of data processing;

[0133] S52: Data compression: After the data integration is completed, a lossless compression algorithm (such as Huffman coding) is used to compress the integrated data. Huffman coding generates the shortest code by constructing an optimal binary tree, ensuring that the data storage space is significantly reduced without losing information.

[0134] S53: Labeling: Labeling the compressed data. Labeling is to assign one or more labels to the data according to the data attributes. The data attributes include source, type and application scenario. The label generation rules are as follows: i =Tag(D i );

[0135] Among them, L i For data item D i Tag is a tag generation function. Tag processing helps to quickly classify and retrieve data.

[0136] S54: Classified storage: According to the optimized classification standards, the labeled data assets are classified and stored. Classified storage can use distributed databases, cloud storage and other technologies to improve data management efficiency and retrieval speed;

[0137] The efficiency of data storage and retrieval is improved by integrating, compressing and labeling highly correlated data sets. Through classified storage, data assets are managed more systematically to ensure efficient use and convenient access to data within the enterprise.

[0138] The data asset visualization in step S6 specifically includes:

[0139] S61: Data distribution map generation: Generate data distribution maps of different dimensions based on classified and stored data assets. The distribution maps can show the distribution of data in different categories, time periods, and regions. The distribution characteristics of data can be intuitively displayed in the form of heat maps, bar charts, pie charts, etc., helping enterprise managers understand the overall overview and distribution trend of data.

[0140] S62: Data association graph generation: Based on the association analysis results, a data association graph is generated. The association graph displays the association and relationship strength between data sets in the form of nodes and edges. The association graph is displayed in the form of a force-directed graph. By simulating the gravitational and repulsive forces in the physical world, it intuitively displays the relationship structure between data nodes and helps discover the potential associations and patterns between data.

[0141] S63: Application of data visualization tools: Use data visualization tools (such as Tableau, Power BI) or enterprise-defined visualization platforms to display the generated data distribution graphs and correlation graphs. Users can use these tools to perform interactive operations, such as filtering, zooming in, and segmenting, to deeply explore patterns and trends in the data and provide intuitive support for enterprise management decisions.

[0142] By generating multi-dimensional data distribution maps and correlation maps, enterprises can intuitively understand and monitor the status of data assets. The application of data visualization tools further enhances the enterprise's insight into data and helps managers make quick data-based decisions, thereby improving the enterprise's business response speed and decision-making quality.

[0143] The optimization process of data asset classification storage in step S5 further includes:

[0144] S55: Distributed storage strategy formulation: Develop a data distribution strategy based on the size, data type, access frequency, and business needs of data assets. The strategy includes selecting a data distribution algorithm (such as hash distribution, range distribution), determining the number of data copies to ensure data redundancy and reliability, and selecting the best storage node to achieve fast data access.

[0145] S56: Data copy management: To improve data accessibility and fault tolerance, multiple data copies are generated. The copy generation solution includes multiple backups of data and distributing the copies in different geographical locations or on different storage nodes to reduce the risk of data loss caused by single point failures and ensure consistency and synchronization of data copies between different nodes.

[0146] S57: Storage node optimization: Storage nodes are optimized and adjusted according to the access frequency and timeliness of data. Data with high access frequency should be stored in high-speed storage nodes (such as SSD) to improve the data reading speed. Data with low access frequency is stored in low-cost storage media (such as HDD or cold storage). Through this optimization, enterprises can find a balance between performance and cost.

[0147] S58: Data access optimization: After data classification and storage are completed, in order to improve data retrieval efficiency, a data access optimization strategy is formulated. The strategy includes establishing indexes to speed up data retrieval, applying caching mechanisms to reduce the pressure of repeated queries on the database, and using data pre-fetching technology to load data that users may need in advance, thereby reducing response time.

[0148] The data asset visualization in step S6 further includes:

[0149] S64, security assessment of visualization results: After generating visualization results, a comprehensive security assessment is conducted to ensure that sensitive data will not be leaked during the visualization process. Sensitive data includes employee identity information, financial data, corporate contract information, and corporate customer information. The security assessment includes classifying the sensitivity level of the data, defining data display permissions at different sensitivity levels, and applying data desensitization technology (such as obfuscation and encrypted display) to ensure data security.

[0150] S65, Data encryption display: For data with high sensitivity (such as financial data), dynamic encryption technology is used in visual display. The encryption display technology includes permission-based encryption level adjustment, that is, displaying data of different granularities according to the user's permission level. The multi-layer encryption mechanism ensures that the data can be effectively protected during transmission and display.

[0151] S66, security log record: During the data visualization process, all access and operations involving sensitive data are recorded in real time. The security log records include user access records, data display records, potential security threat detection, etc. These records provide a complete basis for subsequent security audits and abnormal behavior analysis;

[0152] S67, Security report generation: Based on security assessment and log records, security reports are generated regularly. The reports list in detail all security risks involved in the visualization process, security measures taken, and further improvement suggestions. Security reports can help enterprises detect and respond to potential security threats in a timely manner and ensure the continued security of data.

[0153] The security assessment and protection measures of data visualization ensure the security of sensitive data during the display process. Through encryption technology, permission control and security log recording, enterprises can effectively prevent the risk of data leakage and respond to potential security threats in a timely manner, thereby enhancing the security and compliance of data management.

[0154] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0155] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for combing data assets, characterized in that: The following steps are involved: S1. Preliminary classification of data assets: Preliminary classification of all data assets within the enterprise. The classification criteria include data type, data source, data structure and data storage method. The preliminary classification adopts a rule-based classification algorithm to generate a classification model based on the characteristics and relevance of data assets, and dynamically adjusts and optimizes it according to business needs; S2. Data asset quality assessment: Conduct quality assessment on the data assets after preliminary classification and generate a quality assessment report. The assessment indicators include data integrity, data accuracy, data consistency and data timeliness; S3. Data redundancy detection and optimization: Based on the results of data asset quality assessment, data redundancy detection is performed. The detection content includes data duplication, invalidity and conflict. Data deduplication algorithm and optimization algorithm are used to clean and optimize redundant data. S4. Data correlation analysis: After data redundancy detection and optimization, data assets are analyzed for correlation. The analysis methods include correlation analysis based on graph models and correlation analysis based on statistical models. Data correlation graphs are generated to clarify the relationships between different data sets. S5. Data asset integration and classified storage: Based on the results of data correlation analysis, data fusion algorithms are used to integrate highly correlated data sets into a unified data set, and data compression technology is used to reduce data storage space. The integrated data assets are labeled according to business needs and classified and stored according to optimized classification standards; S6. Data asset visualization: Visualize the classified and stored data assets and generate multi-dimensional display diagrams, including data distribution diagrams and correlation diagrams, to help enterprise managers fully understand and monitor the status of data assets.

2. A method for combing data assets according to claim 1, characterized in that: The preliminary classification of data assets in step S1 specifically includes: S11, data feature extraction: extract multi-dimensional features from various data assets within the enterprise. The multi-dimensional features include data type, data source, data structure and data storage method; S12, classification rule formulation: establishing classification rules based on the extracted features; S13, classification model generation: preliminarily classify the data according to the established classification rules, generate a classification data set, apply the support vector machine algorithm to build a classification model, use the classification data set as training data to train the classification model, use the cross-validation technology to evaluate the model performance, adjust the model parameters according to the validation results, and improve the classification accuracy; S14, classification result output: classify all data assets according to the classification model and generate a classification report. The report content includes the classification results and classification basis.

3. A method for combing data assets according to claim 2, characterized in that: The data asset quality assessment in step S2 specifically includes: S21: Data integrity assessment: Calculate the integrity of the data set, calculated as: Where I is the data integrity percentage, D c is the number of complete data items, D t is the total number of data items; S22: Data accuracy assessment: Calculate the accuracy of the data by comparing it with the reference data set, which is calculated as: Among them, A is the data accuracy percentage, D a is the exact number of data items, D t is the total number of data items; S23: Data consistency assessment: Evaluate the consistency of data between different systems, calculated as: Among them, C is the data consistency percentage, D s is the number of consistent data items, D t is the total number of data items; S24: Data timeliness evaluation: Calculate data timeliness based on timestamps, calculated as: Among them, T is the data timeliness percentage, T n is the current time, T d The latest update time of the data, T o The time when the data was created; S25: Generation of quality assessment report: Integrate various assessment indicators to generate a data asset quality assessment report to provide a basis for subsequent processing.

4. A method for combing data assets according to claim 3, characterized in that: The data redundancy detection and optimization in step S3 specifically includes: S31: Data duplication detection: Use hash algorithm to uniquely encode data items and detect the duplication of data items by generating hash values; S32: Data invalidity detection: Identify invalid or low-value data items based on their frequency of use and relevance in business processes; S33: Data conflict detection: After checking for data duplication and invalidity, check whether there are logical or numerical conflicts between data items; S34: Data optimization: Based on the redundancy detection results, use deduplication and optimization algorithms to clean the data set, delete redundant data, merge or correct conflicting data, and finally generate an optimized data set.

5. A method for combing data assets according to claim 4, characterized in that: The data correlation analysis in step S4 specifically includes: S41: Graph model construction: construct a graph model G(V,E) representing the relationship between data sets, where V represents data nodes and E represents the relationship edge between nodes; S42: Association calculation: Based on the graph model, calculate the association strength between nodes; S43: Statistical model analysis: Use statistical methods to perform correlation analysis on the data set, and the statistical method uses the Pearson correlation coefficient; S44: Data association graph generation: Generate a data association graph based on the results of association calculation and statistical analysis.

6. A method for combing data assets according to claim 5, characterized in that: The data integration and classified storage in step S5 specifically include: S51: Data integration: Based on the results of correlation analysis, the highly correlated data sets are integrated. The principal component analysis method is used in the data integration process to reduce redundant dimensions. S52: Data compression: After the data integration is completed, the integrated data is compressed using a lossless compression algorithm; S53: labeling processing: labeling the compressed data. Labeling is to assign one or more labels to the data according to the attributes of the data. The attributes of the data include source, type and application scenario. S54: Classification storage: Classify and store labeled data assets according to the optimized classification standards.

7. A method for combing data assets according to claim 6, characterized in that: The data asset visualization in step S6 specifically includes: S61: Data distribution graph generation: Generate data distribution graphs of different dimensions based on the classified and stored data assets; S62: Data association graph generation: Based on the association analysis results, a data association graph is generated. The association graph displays the association and relationship strength between data sets in the form of nodes and edges. S63: Application of data visualization tools: Use data visualization tools or customized visualization platforms within the enterprise to display the generated data distribution diagrams and correlation diagrams.

8. A method for combing data assets according to claim 6, characterized in that: The optimization process of data asset classification storage in step S5 further includes: S55: Distributed storage strategy formulation: Develop data distribution strategies based on the scale, data type, access frequency, and business needs of data assets; S56: Data copy management: To improve data accessibility and fault tolerance, multiple data copies are generated. The copy generation solution includes backing up the data multiple times and distributing the copies in different geographical locations or on different storage nodes. S57: Storage node optimization: Optimize and adjust storage nodes based on data access frequency and timeliness; S58: Data access optimization: After data classification and storage are completed, a data access optimization strategy is formulated to improve data retrieval efficiency.

9. A method for combing data assets according to claim 7, characterized in that: The data asset visualization in step S6 further includes: S64, Security assessment of visualization results: After generating visualization results, a comprehensive security assessment is performed to ensure that sensitive data will not be leaked during the visualization process; S65, Data encryption display: For data with high sensitivity level, dynamic encryption technology is used in visual display; S66, Security Logging: During data visualization, all access and operations involving sensitive data are recorded in real time; S67, Security report generation: Generate security reports regularly based on security assessments and log records.

Images