Intelligent analysis and treatment method and system for unstructured data

Abstract

The invention relates to the technical field of data governance, and particularly discloses an intelligent analysis and governance method and system for unstructured data, and the method comprises the steps: determining a data value activity coefficient based on data activity and data blood relationship integrity, determining a cost efficiency coefficient based on a storage cost-benefit ratio and a data redundancy and repetition rate, and carrying out the analysis and governance of the unstructured data. Determining a value efficiency matching degree according to the data value activity coefficient and the cost efficiency coefficient; obtaining a data retention period compliance rate and a life cycle strategy coverage rate of the unstructured data, and determining a compliance risk control coefficient; according to the value efficiency matching degree and the compliance risk control coefficient data life cycle health degree, a treatment opinion corresponding to the data life cycle health degree is output; according to the method, multiple dimensions of value, cost and compliance are integrated, the health degree evaluation model is used for outputting single and visual health degree indexes and graded treatment opinions, a closed loop from evaluation to decision is formed, and the automation and intelligence level of data treatment is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of data governance technology, specifically to a method and system for intelligent parsing and governance of unstructured data. Background Technology

[0002] With the rapid development of big data, artificial intelligence, and cloud computing technologies, the unstructured data accumulated by various industries is experiencing explosive growth. Unstructured data can include images, audio, video, documents, and log files. Due to its diverse formats, unfixed structure, and complex semantics, it is difficult to effectively manage and analyze using traditional relational databases or structured data processing tools. According to industry reports, unstructured data already accounts for more than 80% of the total data volume of enterprises, becoming an important and indispensable part of data assets.

[0003] However, the rapid growth of unstructured data has also brought a series of governance challenges. First, the value of data is difficult to quantify, and a large amount of data remains idle due to the lack of effective activity assessment mechanisms, resulting in resource waste. Second, storage costs are constantly rising, and data redundancy and duplicate storage are prominent issues, seriously affecting storage efficiency and economy. Third, the unclear data lineage makes data traceability difficult, affecting data credibility and usability. In addition, the lack or inadequacy of data lifecycle management strategies makes data retention, archiving, and destruction lack compliance basis, posing legal and compliance risks.

[0004] Currently, while some data governance tools and platforms exist in the market, most still focus on the management of structured data or only provide basic data classification and tagging functions, lacking the ability to intelligently analyze and comprehensively govern unstructured data throughout its entire lifecycle. Existing methods often rely on manually formulated strategies, which are inefficient and difficult to adapt to the dynamic nature of data changes, failing to achieve synergistic optimization of data value and cost efficiency.

[0005] Therefore, there is an urgent need for a method and system that can systematically assess the value activity, cost efficiency and compliance of unstructured data, and intelligently output governance recommendations based on the assessment results, so as to achieve refined, automated and intelligent management of data assets. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for intelligent parsing and governance of unstructured data, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A method for intelligent parsing and governance of unstructured data, the method comprising: Establish a connection channel with the log database based on preset permissions, and determine the data activity and data lineage integrity of unstructured data based on the log database; To assess the cost-effectiveness of storing unstructured data and the levels of data redundancy and duplication; The data value activity coefficient is determined based on data activity and data lineage integrity; the cost efficiency coefficient is determined based on storage cost-effectiveness ratio and data redundancy and duplication rate; and the value efficiency matching degree is determined based on the data value activity coefficient and cost efficiency coefficient. Obtain the data retention period compliance rate and lifecycle strategy coverage rate of unstructured data to determine the compliance risk control coefficient; Based on the value-efficiency matching degree and compliance risk control coefficient, the data lifecycle health is evaluated, and corresponding governance opinions are output.

[0008] Furthermore, the step of determining the data value activity coefficient based on data activity and data lineage integrity includes: The current data activity level is processed by max-min normalization to obtain the data activity index; The ratio of the current number of traced bloodline nodes to the total number of traced bloodline nodes is calculated to obtain the data bloodline integrity index. According to the value fusion rule, the data activity index and the data lineage integrity index are integrated to generate the data value activity coefficient. The value fusion rule defines the mapping relationship from data usage frequency and data lineage quality to data value activity. The data value activity coefficient is positively correlated with both the data activity index and the data lineage integrity index.

[0009] Furthermore, the step of determining the cost efficiency coefficient based on the storage cost-effectiveness ratio and data redundancy and duplication rate includes: The storage cost-benefit ratio is obtained by comparing the total storage cost with the data access revenue. The storage cost-benefit ratio is obtained by performing maximum-min normalization on the current storage cost-benefit ratio. According to the economic mapping rule, the cost efficiency coefficient is determined by the storage cost-effectiveness ratio index and the data redundancy and duplication rate; wherein, the economic mapping rule is constructed such that the cost efficiency coefficient is negatively correlated with both the storage cost-effectiveness ratio index and the data redundancy and duplication rate.

[0010] Furthermore, the step of determining the value efficiency matching degree based on the data value activity coefficient and cost efficiency coefficient includes: Based on the data value activity coefficient and cost efficiency coefficient, the value efficiency matching degree is output through the matching model; The matching model is configured to process the data value activity coefficient and the cost efficiency coefficient through a balance function to output the value efficiency matching degree; the balance function is constructed to respond simultaneously to the synergy level of the activity coefficient and the efficiency coefficient and the degree of difference between them.

[0011] Furthermore, the steps for obtaining the data retention period compliance rate and lifecycle strategy coverage rate of unstructured data, and determining the compliance risk control coefficient, include: The data retention period compliance rate is obtained by comparing the amount of data that meets the retention period requirements with the total amount of data. The lifecycle strategy coverage is obtained by comparing the number of data types for which strategies have been defined with the total number of data types for which strategies should be defined. Based on the data retention period compliance rate and the lifecycle strategy coverage rate, the compliance risk control coefficient is calculated through risk control aggregation rules; the risk control aggregation rules are used to integrate the evaluation results of the compliance rate and the coverage rate, and the compliance risk control coefficient is positively correlated with both the data retention period compliance rate and the lifecycle strategy coverage rate.

[0012] Furthermore, the step of outputting governance opinions corresponding to the data lifecycle health based on the value efficiency matching degree and compliance risk control coefficient includes: Based on preset association rules, the value efficiency matching degree is combined with the compliance risk control coefficient to output the data lifecycle health degree; The association rule defines the joint influence mechanism of the matching degree and the risk control coefficient on the data health status. The data lifecycle health is positively correlated with the value efficiency matching degree and the compliance risk control coefficient.

[0013] The present invention also provides an intelligent analysis and governance system for unstructured data, the system comprising: The log data analysis module is used to establish a connection channel with the log database based on preset permissions, and to determine the data activity and data lineage integrity of unstructured data based on the log database. The data cost analysis module is used to obtain the storage cost-effectiveness ratio and data redundancy and duplication rate of unstructured data; The value-cost matching module is used to determine the data value activity coefficient based on data activity and data lineage integrity, determine the cost efficiency coefficient based on storage cost-benefit ratio and data redundancy and duplication rate, and determine the value-efficiency matching degree based on the data value activity coefficient and cost efficiency coefficient. The risk control coefficient determination module is used to obtain the data retention period compliance rate and lifecycle strategy coverage rate of unstructured data, and to determine the compliance risk control coefficient. The data health assessment module is used to assess the health of the data lifecycle based on the value-efficiency matching degree and compliance risk control coefficient, and output governance opinions corresponding to the data lifecycle health.

[0014] Furthermore, the value-cost matching module includes: The activity analysis unit is used to perform max-min normalization on the current data activity level to obtain the data activity index. The data lineage analysis unit is used to process the ratio of the number of currently tracked lineage nodes to the total number of lineage nodes that should be tracked, and obtain the data lineage integrity index. The activity analysis unit is used to integrate the data activity index and the data lineage integrity index according to the value fusion rule to generate the data value activity coefficient; the value fusion rule defines the mapping relationship from data usage frequency and data lineage quality to data value activity, and the data value activity coefficient is positively correlated with both the data activity index and the data lineage integrity index.

[0015] Furthermore, the value-cost matching module also includes: The cost-benefit ratio acquisition unit is used to process the ratio of total storage cost to data access revenue value to obtain the storage cost-benefit ratio; The normalization processing unit is used to perform maximum-min normalization on the current storage cost-benefit ratio to obtain the storage cost-benefit ratio index. The cost efficiency analysis unit is used to determine the cost efficiency coefficient based on the storage cost-effectiveness ratio index and the data redundancy and duplication rate according to the economic mapping rule; wherein the economic mapping rule is constructed such that the cost efficiency coefficient is negatively correlated with both the storage cost-effectiveness ratio index and the data redundancy and duplication rate.

[0016] Furthermore, the value-cost matching module also includes: The matching execution unit is used to output the value-efficiency matching degree based on the data value activity coefficient and cost efficiency coefficient through the matching model. The matching model is configured to process the data value activity coefficient and the cost efficiency coefficient through a balance function to output the value efficiency matching degree; the balance function is constructed to respond simultaneously to the synergy level of the activity coefficient and the efficiency coefficient and the degree of difference between them.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention integrates multiple dimensions of value, cost, and compliance, and uses a health assessment model to output a single, intuitive health indicator and tiered governance recommendations, forming a closed loop from assessment to decision-making. This significantly improves the automation and intelligence of data governance and overcomes the problems of traditional methods that rely on human experience and are inefficient. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.

[0019] Figure 1 The overall flowchart of the intelligent parsing and governance method for unstructured data is shown.

[0020] Figure 2 The diagram shows the structure of an intelligent analysis and governance system for unstructured data. Detailed Implementation

[0021] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0022] Figure 1 This is a flowchart illustrating the overall process of an intelligent parsing and governance method and system for unstructured data. In this embodiment of the invention, an intelligent parsing and governance method for unstructured data includes: Step S100: Establish a connection channel with the log database based on preset permissions, and determine the data activity and data lineage integrity of unstructured data based on the log database; Step S200: Obtain the storage cost-effectiveness ratio and data redundancy and duplication rate of unstructured data; Step S300: Determine the data value activity coefficient based on data activity and data lineage integrity; determine the cost efficiency coefficient based on storage cost-benefit ratio and data redundancy and duplication rate; and determine the value efficiency matching degree based on the data value activity coefficient and cost efficiency coefficient. Step S400: Obtain the data retention period compliance rate and lifecycle strategy coverage rate of unstructured data to determine the compliance risk control coefficient; Step S500: Based on the value efficiency matching degree and compliance risk control coefficient, the data lifecycle health is determined, and governance opinions corresponding to the data lifecycle health are output.

[0023] In one embodiment of the technical solution of this invention, a data processing procedure for unstructured data is provided. Based on data activity and data lineage integrity, a data value activity coefficient is output through a data value activity model. In practical applications, this model can be implemented using expert scoring methods, such as independent scoring by business departments based on data activity and data lineage integrity, followed by weighted calculation of the coefficient. Alternatively, a statistical regression model can be established through historical access log analysis to predict the value coefficient. Data activity can be obtained by statistically analyzing file access frequency per unit time using system logs, while data lineage integrity can be determined by manually verifying data source records or reviewing metadata documents. Based on storage cost-effectiveness ratio and data redundancy and duplication rate, a cost efficiency coefficient is output through a cost efficiency model. The cost efficiency model is used to output the cost efficiency coefficient, and its implementation includes manual evaluation based on a cost-effectiveness analysis table. For example, the storage cost-effectiveness ratio is obtained by comparing the total storage cost with the business revenue value, and the redundancy rate is calculated by combining the redundancy data ratio reported by the storage system. Then, a coefficient is generated using a simple averaging method. The storage cost-effectiveness ratio can be obtained from the financial system by acquiring storage fees and linking them to the business revenue report. Data redundancy and duplication rate can be determined by identifying duplicate files using file hash value comparison tools.

[0024] Based on the data value activity coefficient and cost efficiency coefficient, the matching model outputs the data value-cost efficiency matching degree. In practical applications, this model can be implemented by constructing a decision matrix, such as mapping the value coefficient and cost coefficient to a two-dimensional coordinate system to calculate the distance, or using predefined threshold rules to divide the matching level according to the coefficient range.

[0025] Based on the data retention period compliance rate and lifecycle strategy coverage rate, a compliance risk control coefficient is output through a compliance risk control model. The data retention period compliance rate can be obtained by scanning whether the data retention period complies with regulatory requirements using compliance inspection tools, while the lifecycle strategy coverage rate can be determined by reviewing strategy documents and statistically analyzing the proportion of data types for which strategies have been formulated. This model can be implemented by the legal team scoring using a risk assessment questionnaire.

[0026] Based on the data value-cost efficiency matching degree and compliance risk control coefficient, a health assessment model outputs the data lifecycle health degree and corresponding governance opinions. This model can calculate the health degree using a weighted summation method, such as adding the matching degree and compliance coefficient at a fixed ratio and then outputting tiered governance opinions based on threshold ranges, or generating specific opinions through expert meetings and matching with a predefined governance measures library. This application, through the integration of the above multi-dimensional model system, achieves a systematic quantitative assessment of unstructured data, solving problems in the background technology such as difficulty in quantifying data value, high storage costs, unclear data lineage, and compliance risks caused by the lack of lifecycle management strategies. It enables data governance decisions to be automatically output based on quantitative indicators, avoiding the inefficient model of relying on manual strategy formulation.

[0027] Therefore, this method achieves a unified solution for quantifying the value of unstructured data, controlling costs, tracing lineage, and managing compliance by systematically constructing a cascaded processing flow of a data value activity model, a cost efficiency model, a matching model, a compliance risk control model, and a health assessment model. Among these solutions, the generation of the data value activity coefficient effectively overcomes the problem of misjudgment of value caused by a single indicator; the output of the cost efficiency coefficient specifically alleviates the waste of storage resources; the calculation of the matching degree ensures a dynamic balance between data value and storage costs; the quantification of the compliance risk control coefficient avoids legal risks caused by the lack of lifecycle strategies; and finally, the output of governance opinions driven by the health assessment results transforms data asset management from an inefficient model relying on manual experience to an automated and intelligent closed-loop governance model, significantly improving the refinement level and decision reliability of unstructured data governance.

[0028] It should be noted that the inputs to the above processing are four parameters: data activity, data lineage integrity, storage cost-effectiveness ratio, and data redundancy and duplication rate. The specific acquisition process has been mentioned above. To facilitate understanding of the above scheme by those skilled in the art, these four parameters will be further explained. Data activity indicates the frequency and intensity of data usage and access in the storage system, including read counts, write counts, modification counts, and transfer counts. Its physical meaning is the most direct indicator of whether data is being used. Analyzing this parameter can determine whether data is "hot" or "cold," avoiding storing cold data in expensive high-speed storage, reducing costs, and preventing the deletion of data that is still actively used. Data lineage integrity indicates whether the source, processing, change path, and destination of a piece of data are complete and traceable. Its physical meaning is to determine whether data is "isolated," providing a basis for data deletion and archiving. The ability to trace the source and avoid accidental deletion of critical data is crucial. Since the technical solution of this invention deals with unstructured data, the lineage analysis process is somewhat difficult. It often requires tracing the file generation path, recording the process of file movement, copying and processing between systems, and extracting file metadata (such as shooting time, author and source system) to make a comprehensive judgment. There are relevant judgment rules in the existing technology, which will not be limited here. In practical applications, most of the time, the management party sets some weights, counts these data, and normalizes them into a single value to be used as the data lineage integrity. The storage cost-effectiveness ratio represents the economic efficiency between the "business value" generated by storing a certain type of data and the storage resources occupied. It is a key metric for the governance of unstructured data storage. Data redundancy and duplication rate represent the proportion of duplicate files, similar content or invalid copies in the dataset. The practical meaning of these two parameters is very simple, and their determination process also has relevant solutions in the existing technology. It is not complicated for those skilled in the art, and will not be limited here.

[0029] Based on four parameters—known data activity, data lineage integrity, storage cost-effectiveness ratio, and data redundancy and duplication rate—data analysis is performed, generating extremely comprehensive governance recommendations.

[0030] Regarding step S300, as a preferred embodiment of the technical solution of the present invention, the step of determining the data value activity coefficient based on data activity and data lineage integrity includes: The current data activity level is processed by max-min normalization to obtain the data activity index; The ratio of the current number of traced bloodline nodes to the total number of traced bloodline nodes is calculated to obtain the data bloodline integrity index. According to the value fusion rule, the data activity index and the data lineage integrity index are integrated to generate the data value activity coefficient. The value fusion rule defines the mapping relationship from data usage frequency and data lineage quality to data value activity. The data value activity coefficient is positively correlated with both the data activity index and the data lineage integrity index.

[0031] Furthermore, the step of determining the cost efficiency coefficient based on the storage cost-effectiveness ratio and data redundancy and duplication rate includes: The storage cost-benefit ratio is obtained by comparing the total storage cost with the data access revenue. The storage cost-benefit ratio is obtained by performing maximum-min normalization on the current storage cost-benefit ratio. According to the economic mapping rule, the cost efficiency coefficient is determined by the storage cost-effectiveness ratio index and the data redundancy and duplication rate; wherein, the economic mapping rule is constructed such that the cost efficiency coefficient is negatively correlated with both the storage cost-effectiveness ratio index and the data redundancy and duplication rate.

[0032] Specifically, the step of determining the value efficiency matching degree based on the data value activity coefficient and cost efficiency coefficient includes: Based on the data value activity coefficient and cost efficiency coefficient, the value efficiency matching degree is output through the matching model; The matching model is configured to process the data value activity coefficient and the cost efficiency coefficient through a balance function to output the value efficiency matching degree; the balance function is constructed to respond simultaneously to the synergy level of the activity coefficient and the efficiency coefficient and the degree of difference between them.

[0033] The above content corresponds to the entire process of step S300, specifically the normalization process of the four parameters. It consists of three steps. The first step is to output the data value activity coefficient based on data activity and data lineage integrity through the data value activity model. The specific steps are as follows: The current data activity is processed by max-min normalization to obtain the data activity index. Max-min normalization is a standardization method that linearly transforms the original data to the [0,1] interval. It can be achieved by linearly mapping the minimum and maximum values ​​of historical data. Its purpose is to eliminate the differences in the units and ranges of activity from different data sources and ensure the objectivity and consistency of the assessment.

[0034] The ratio of the number of currently tracked bloodline nodes to the total number of bloodline nodes to be tracked is used to obtain the data bloodline integrity index. The ratio processing can be understood as calculating the proportional relationship between two values. Specifically, it can be implemented by direct division. Its purpose is to quantify the bloodline integrity measure into a dimensionless index, which is convenient for subsequent fusion calculation.

[0035] According to the value fusion rule, the data activity index and the data lineage integrity index are integrated to generate the data value activity coefficient. The value fusion rule defines the mapping relationship from data usage frequency and data lineage quality to data value activity. The data value activity coefficient is positively correlated with both the data activity index and the data lineage integrity index. In practical applications, the calculation formula for the data value activity coefficient can be understood as a weighted average model, which can be implemented using software algorithms configured with weighting coefficients in the data processing engine. The purpose is to achieve this through the weighting coefficients... The contribution ratios of activity level and lineage integrity can be dynamically adjusted to adapt to the needs of different business scenarios.

[0036] The value fusion rule specifically involves importing the data activity index and data lineage integrity index into the formula. To obtain the data value activity coefficient , The value ranges from 0 to 1. A larger value indicates more frequent data access and usage, a more complete data source traceability chain, and higher potential business value. This is a data activity index. This is a data lineage integrity index. This is the data activity weighting coefficient. The value range is 0-1.

[0037] The above scheme normalizes data activity to generate a standardized activity index, eliminating dimensional differences in the original data. It then calculates the coverage ratio of lineage nodes to obtain a lineage integrity index, transforming ambiguous lineage relationships into measurable indicators. Finally, the two indices are fused using a weighted formula to output a data value activity coefficient. This sequential processing mechanism ensures that the original data is gradually transformed into objective quantitative indicators, avoiding evaluation biases caused by differences in data scale or access patterns, and providing a reliable input basis for data value-cost efficiency matching. In unstructured data storage systems, data activity can be measured based on file access frequency, and the data activity index is obtained through max-min normalization. The data lineage integrity index can be calculated based on the ratio of tracked lineage nodes to the total number of nodes that should be tracked. The data value activity coefficient is calculated using a weighted formula, where the weighting coefficients are... The data governance platform can be dynamically configured according to business needs. For example, in a document management system, the platform can call the data processing engine to perform normalization, ratio calculation, and weighted fusion operations.

[0038] In practical applications, the above scheme achieves objective quantification of the data value activity coefficient, avoids evaluation bias caused by subjective judgment or inconsistent methods, ensures the accuracy of data value-cost efficiency matching degree calculation, and thus improves the accuracy and reliability of unstructured data governance.

[0039] The second step is to output a cost efficiency coefficient based on the storage cost-effectiveness ratio and data redundancy and duplication rate through a cost efficiency model. The specific steps are as follows: The storage cost-benefit ratio is obtained by comparing the total storage cost with the data access revenue. Specifically, the storage cost-benefit ratio is obtained by dividing the total cost of the storage system by the business value revenue brought by data access. The purpose is to transform absolute cost and relative revenue into a dimensionless ratio indicator to avoid the incomparability of the original values ​​due to differences in units or magnitudes.

[0040] The storage cost-benefit ratio is obtained by performing maximum-min normalization on the current storage cost-benefit ratio. Maximum-min normalization is a method of linearly transforming the data to the [0,1] interval. It can be achieved by using a linear transformation based on historical datasets to dynamically determine the extreme points. Its purpose is to eliminate numerical fluctuations caused by differences in data scale under different business scenarios and ensure that the indicators are comparable under a unified scale.

[0041] Based on an economic mapping rule, the cost efficiency coefficient is determined by the storage cost-effectiveness ratio index and the data redundancy and duplication rate; wherein, the economic mapping rule is constructed such that the cost efficiency coefficient is negatively correlated with both the storage cost-effectiveness ratio index and the data redundancy and duplication rate; the cost efficiency coefficient is a single evaluation index that integrates the storage cost-effectiveness ratio index and the data redundancy and duplication rate through a weighted combination formula, which can be implemented by dynamically adjusting the weight coefficients, with the aim of making the output coefficient strictly positively correlated with storage economy, facilitating subsequent model integration.

[0042] The economic mapping rule is to import the storage cost-effectiveness ratio index and data redundancy and duplication rate into the formula. Obtain the cost efficiency coefficient , The value ranges from 0 to 1. A larger value indicates better cost-effectiveness in data storage and less redundant or duplicated data, resulting in higher economic efficiency in storage. This is a storage cost-effectiveness index. To reduce data redundancy and duplication, The value range is 0-1. The storage cost-effectiveness ratio is a weighted coefficient. The value range is 0-1.

[0043] The above scheme transforms physical costs and business value into a dimensionless ratio by comparing total storage costs with data access benefits. It then applies maximum-minimum normalization to the storage cost-effectiveness ratio, compressing it to a standard range based on the actual distribution of the current dataset to eliminate dimensional differences. Finally, the normalized storage cost-effectiveness ratio index, along with data redundancy and duplication rates, is incorporated into a weighted formula. The mathematical structure of subtracting the weighted sum from 1 ensures a positive correlation between the cost efficiency coefficient and storage economics. Because the normalization process places the storage cost-effectiveness ratio index and data redundancy rate on the same numerical scale, dimensional interference is avoided during weighted calculations, thus guaranteeing the accurate generation of the cost efficiency coefficient and providing a reliable input for evaluating the data value-cost efficiency matching.

[0044] In an enterprise document management system, the total storage cost is first compared to the data access revenue to obtain the storage cost-benefit ratio. Then, based on historical data ranges, this ratio is normalized using a maximum-minimum method, converting it into a storage cost-benefit ratio index within the [0,1] interval. Simultaneously, data redundancy and duplication rates are measured using hash comparison technology. Finally, the normalized index and redundancy rate are substituted into a weighted formula to calculate the cost efficiency coefficient. For example, when the normalized storage cost-benefit ratio is at a moderate level and the data redundancy rate is low, the calculated cost efficiency coefficient is higher, intuitively reflecting a good state of storage economics.

[0045] The third step is the setting and application of the matching model, which is configured to: process the data value activity coefficient and the cost efficiency coefficient through a balance function to output the data value-cost efficiency matching degree; the balance function is constructed to respond simultaneously to the synergy level of the activity coefficient and the efficiency coefficient and the degree of difference between the two. Specifically, the matching model is as follows: ,in, For data value-cost efficiency matching, The value ranges from 0 to 1. The larger the value, the better the balance and synergy between the high value of the data and the low cost of storage. The data value activity coefficient. The cost efficiency coefficient. To minimize division by zero errors, it is usually taken as a very small constant. .

[0046] The geometric mean term refers to the square root operation of the product of the data value activity coefficient and the cost efficiency coefficient. It can be implemented using a nonlinear combination function. Its purpose is to ensure that both data value activity and cost efficiency must be at a high level simultaneously to obtain a high basic matching value, thereby avoiding a single indicator dominating the evaluation result. The difference adjustment term refers to a mechanism that dynamically corrects the absolute difference between the value activity coefficient and the cost efficiency coefficient relative to their sum. It can be implemented using exponential decay or other nonlinear decay methods. Its purpose is to quantify the degree of deviation between value and cost efficiency and effectively suppress the overestimation of the matching degree when the difference between the indicators is significant. In practical applications, the minimum constant refers to a positive threshold that approaches zero. Its purpose is to prevent calculation anomalies caused by the denominator being zero when both the value activity coefficient and the cost efficiency coefficient approach zero, and to ensure the robustness of the model under boundary conditions.

[0047] For the practical application of the above scheme, the system first inputs the data value activity coefficient and cost efficiency coefficient into the geometric mean calculation unit to generate a basic matching value. Then, the difference calculation unit obtains the absolute difference between the two coefficients and performs a ratio calculation with the sum of the two coefficients and a minimum constant to generate a correction factor. Finally, the multiplication unit multiplies the basic matching value by the correction factor to output the data value-cost efficiency matching degree. When the two coefficient values ​​are close, the correction factor is close to 1, and the matching degree is mainly determined by the geometric mean. However, when the two coefficients differ significantly, the correction factor decreases significantly, resulting in a reasonable suppression of the matching degree, thus truly reflecting the synergistic state of value and cost efficiency.

[0048] Regarding step S400, as a preferred embodiment of the technical solution of the present invention, the step of obtaining the data retention period compliance rate and lifecycle strategy coverage rate of unstructured data, and determining the compliance risk control coefficient includes: The data retention period compliance rate is obtained by comparing the amount of data that meets the retention period requirements with the total amount of data. The lifecycle strategy coverage is obtained by comparing the number of data types for which strategies have been defined with the total number of data types for which strategies should be defined. Based on the data retention period compliance rate and the lifecycle strategy coverage rate, the compliance risk control coefficient is calculated through risk control aggregation rules; the risk control aggregation rules are used to integrate the evaluation results of the compliance rate and the coverage rate, and the compliance risk control coefficient is positively correlated with both the data retention period compliance rate and the lifecycle strategy coverage rate.

[0049] The data retention period compliance rate is obtained by comparing the amount of data that meets the retention period requirements with the total amount of data. For example, metadata scanning and timestamp comparison are used to determine whether the data is within the compliance period. The purpose is to transform regulatory compliance into a measurable numerical indicator and avoid the ambiguity of manual verification.

[0050] The lifecycle strategy coverage rate is obtained by comparing the number of data types for which strategies have been established with the total number of data types for which strategies should be established. For example, by comparing the strategy configuration database with the data type list, the purpose is to transform the problem of missing strategies into quantifiable statistical indicators and improve the objectivity of strategy completeness assessment.

[0051] Based on the data retention period compliance rate and the lifecycle strategy coverage rate, the compliance risk control coefficient is calculated through a risk control aggregation rule; the risk control aggregation rule is used to integrate the evaluation results of the compliance rate and the coverage rate, and the compliance risk control coefficient is positively correlated with both the data retention period compliance rate and the lifecycle strategy coverage rate; The calculation method is as follows: Data retention period compliance rate and lifecycle strategy coverage rate are imported into the formula. Obtain compliance risk control coefficient , The value ranges from 0 to 1. A higher value indicates better compliance with regulations regarding data retention periods and a more comprehensive lifecycle management strategy, resulting in lower compliance risks. For data retention compliance rate, for lifecycle strategy coverage, The data retention period compliance rate is weighted by a coefficient. The value range is 0-1.

[0052] The above scheme first calculates the data retention period compliance rate by comparing the amount of data meeting retention period requirements with the total amount of data, objectively reflecting the degree of regulatory compliance. Second, it calculates the lifecycle strategy coverage rate by comparing the number of data types for which strategies have been established with the total number of data types for which strategies should be established, quantifying the scope of strategy implementation. Finally, the two indicators are combined using a weighted formula to form a compliance risk control coefficient, creating a synergistic quantitative mechanism between regulatory compliance and strategy completeness. The data retention period compliance rate weighting coefficient is a parameter used to adjust the importance of different factors in decision-making; it can be assigned values ​​through preset strategies or dynamic algorithms. Its purpose is to avoid the limitations of single-dimensional assessment, ensuring that health assessments simultaneously cover data retention compliance status and strategy management completeness, thereby providing a systematic and reproducible risk assessment basis for governance decisions.

[0053] Specifically, in the data governance scenario of the financial industry, the system automatically scans the customer transaction record repository, compares the metadata timestamps to calculate the percentage of data that meets the 5-year retention period requirement, and uses this percentage as the data retention period compliance rate. Simultaneously, it compares the data types of the configured lifecycle strategies (such as transaction logs and customer information) with the total number of types that should be configured, calculating the lifecycle strategy coverage rate. Finally, it sets up the system according to regulatory priorities. The value is 0.7. Input the two indicators into the formula to calculate the compliance risk control coefficient.

[0054] Regarding step S500, as a preferred embodiment of the technical solution of the present invention, the step of outputting governance opinions corresponding to the data lifecycle health based on the value efficiency matching degree and compliance risk control coefficient data lifecycle health includes: Based on preset association rules, the value efficiency matching degree is combined with the compliance risk control coefficient to output the data lifecycle health degree; The association rule defines the joint influence mechanism of the matching degree and the risk control coefficient on the data health status. The data lifecycle health is positively correlated with the value efficiency matching degree and the compliance risk control coefficient.

[0055] The above content describes the final comprehensive application and output process of all data, which is essentially a health assessment model configured as follows: Based on preset association rules, the data value-cost efficiency matching degree is combined with the compliance risk control coefficient to output the data lifecycle health degree; The association rule defines the joint influence mechanism of the matching degree and the risk control coefficient on the health degree. The data lifecycle health degree is positively correlated with the data value-cost efficiency matching degree and the compliance risk control coefficient. Specifically, the health assessment model is as follows: ,in, For the health of the data lifecycle, For data value-cost efficiency matching, For compliance and risk control coefficients, This is the matching degree weighting coefficient. The value range is 0-1; The health assessment model refers to a mathematical framework used to quantitatively and comprehensively assess the health status of data throughout its lifecycle. It can be implemented using a linear weighted fusion algorithm, dynamically adjusting weight parameters to adapt to the governance priorities of different organizations; the matching degree weight coefficient... This can be understood as an adjustable parameter reflecting the relative importance of data value, cost efficiency, and compliance risk control. It can be preset to a fixed value or optimized in real time based on business scenarios, with the aim of making the health assessment results more aligned with actual governance goals. The health level classification mechanism refers to a classification method that discretizes continuous health values ​​into clearly defined intervals. It can adopt equal-interval threshold settings or statistical division based on historical data distribution, with the aim of transforming abstract values ​​into intuitively identifiable health status levels. The governance opinions corresponding to each health level refer to standardized operating guidelines preset for a specific health level. These guidelines can include specific action instructions such as strategy optimization and data cleaning, with the aim of providing directly executable decision-making basis for data governance.

[0056] Health status classification and corresponding governance recommendations: This indicates that the existing strategy will be maintained, continuous monitoring and optimization will be carried out, and in-depth exploration of data value will be conducted. This indicates a good performance, with localized optimization of cost-effectiveness and improved archiving efficiency for low-activity data. [0.5, 0.7) indicates general, revising the lifecycle strategy, strengthening data deduplication, and completing the lineage relationship; This indicates a poor performance, requiring the enforcement of archiving strategies, a data cleanup campaign, and the reconstruction of the metadata system. This indicates a serious situation, prompting the initiation of an emergency remediation project, a comprehensive restructuring of the lifecycle management system, and the introduction of external expert support.

[0057] In one example of the technical solution of this invention, the above solution generates a continuous data lifecycle health value by inputting the data value-cost efficiency matching degree and compliance risk control coefficient into the health assessment model for weighted calculation. Then, based on a preset threshold range, the health value is divided into five discrete levels, each strictly bound to differentiated governance recommendations. This design ensures that the health assessment is entirely based on objective data indicators rather than human experience, making the assessment results repeatable and consistent. The introduction of the matching degree weight coefficient λ allows the system to flexibly adjust the assessment focus according to organizational needs; for example, in cost-sensitive scenarios, the λ value can be increased to strengthen the value-cost efficiency weight. The discretization of health levels transforms complex continuous assessment results into clear status classifications, with each classification associated with specific governance actions, thereby directly converting the assessment results into actionable governance instructions and avoiding execution deviations caused by generalized recommendations.

[0058] Specifically, after the system receives the input of data value-cost efficiency matching degree and compliance risk control coefficient, it calculates the data lifecycle health degree to be 0.82. The system automatically determines that it falls within the range of [0.7, 0.9), outputs the "good" level label, and generates the corresponding governance opinion "locally optimize cost-effectiveness ratio and improve the archiving efficiency of low-activity data", guiding the operation and maintenance personnel to implement archiving strategy optimization for low-activity data without making large-scale adjustments to the overall strategy.

[0059] By combining the above-mentioned integrated data application solutions, this invention achieves an objective and quantitative assessment of the health of the data lifecycle, eliminating assessment bias caused by subjective judgment. At the same time, through structured hierarchical classification and precise matching of governance opinions, data governance actions can implement differentiated interventions for different health levels, effectively improving the refinement and execution efficiency of unstructured data governance.

[0060] Figure 2 A structural diagram of an intelligent unstructured data parsing and governance system is shown. In a preferred embodiment of the technical solution of the present invention, an intelligent unstructured data parsing and governance system is also provided, the system 10 comprising: Log data analysis module 11 is used to establish a connection channel with the log database based on preset permissions, and to determine the data activity and data lineage integrity of unstructured data based on the log database. Data cost analysis module 12 is used to obtain the storage cost-effectiveness ratio of unstructured data and the data redundancy and duplication rate; The value-cost matching module 13 is used to determine the data value activity coefficient based on data activity and data lineage integrity, determine the cost efficiency coefficient based on storage cost-benefit ratio and data redundancy and duplication rate, and determine the value efficiency matching degree based on the data value activity coefficient and cost efficiency coefficient. The risk control coefficient determination module 14 is used to obtain the data retention period compliance rate and lifecycle strategy coverage rate of unstructured data to determine the compliance risk control coefficient. The data health assessment module 15 is used to assess the health of the data lifecycle based on the value-efficiency matching degree and compliance risk control coefficient, and to output governance opinions corresponding to the health of the data lifecycle.

[0061] Furthermore, the value-cost matching module 13 includes: The activity analysis unit is used to perform max-min normalization on the current data activity level to obtain the data activity index. The data lineage analysis unit is used to process the ratio of the number of currently tracked lineage nodes to the total number of lineage nodes that should be tracked, and obtain the data lineage integrity index. The activity analysis unit is used to integrate the data activity index and the data lineage integrity index according to the value fusion rule to generate the data value activity coefficient; the value fusion rule defines the mapping relationship from data usage frequency and data lineage quality to data value activity, and the data value activity coefficient is positively correlated with both the data activity index and the data lineage integrity index.

[0062] Specifically, the value-cost matching module 13 further includes: The cost-benefit ratio acquisition unit is used to process the ratio of total storage cost to data access revenue value to obtain the storage cost-benefit ratio; The normalization processing unit is used to perform maximum-min normalization on the current storage cost-benefit ratio to obtain the storage cost-benefit ratio index. The cost efficiency analysis unit is used to determine the cost efficiency coefficient based on the storage cost-effectiveness ratio index and the data redundancy and duplication rate according to the economic mapping rule; wherein the economic mapping rule is constructed such that the cost efficiency coefficient is negatively correlated with both the storage cost-effectiveness ratio index and the data redundancy and duplication rate.

[0063] Furthermore, the value-cost matching module 13 also includes: The matching execution unit is used to output the value-efficiency matching degree based on the data value activity coefficient and cost efficiency coefficient through the matching model. The matching model is configured to process the data value activity coefficient and the cost efficiency coefficient through a balance function to output the value efficiency matching degree; the balance function is constructed to respond simultaneously to the synergy level of the activity coefficient and the efficiency coefficient and the degree of difference between them.

[0064] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for intelligent parsing and governance of unstructured data, characterized in that, The method includes: Establish a connection channel with the log database based on preset permissions, and determine the data activity and data lineage integrity of unstructured data based on the log database; To assess the cost-effectiveness of storing unstructured data and the levels of data redundancy and duplication; The data value activity coefficient is determined based on data activity and data lineage integrity; the cost efficiency coefficient is determined based on storage cost-effectiveness ratio and data redundancy and duplication rate; and the value efficiency matching degree is determined based on the data value activity coefficient and cost efficiency coefficient. Obtain the data retention period compliance rate and lifecycle strategy coverage rate of unstructured data to determine the compliance risk control coefficient; Based on the value-efficiency matching degree and compliance risk control coefficient, the data lifecycle health is evaluated, and corresponding governance opinions are output.

2. The method for intelligent parsing and governance of unstructured data according to claim 1, characterized in that, The steps for determining the data value activity coefficient based on data activity and data lineage integrity include: The current data activity level is processed by max-min normalization to obtain the data activity index; The ratio of the current number of traced bloodline nodes to the total number of traced bloodline nodes is calculated to obtain the data bloodline integrity index. According to the value fusion rule, the data activity index and the data lineage integrity index are integrated to generate the data value activity coefficient. The value fusion rule defines the mapping relationship from data usage frequency and data lineage quality to data value activity. The data value activity coefficient is positively correlated with both the data activity index and the data lineage integrity index.

3. The method for intelligent parsing and governance of unstructured data according to claim 2, characterized in that, The steps for determining the cost efficiency coefficient based on storage cost-effectiveness ratio and data redundancy and duplication rate include: The storage cost-benefit ratio is obtained by comparing the total storage cost with the data access revenue. The storage cost-benefit ratio is obtained by performing maximum-min normalization on the current storage cost-benefit ratio. According to the economic mapping rule, the cost efficiency coefficient is determined by the storage cost-effectiveness ratio index and the data redundancy and duplication rate; wherein, the economic mapping rule is constructed such that the cost efficiency coefficient is negatively correlated with both the storage cost-effectiveness ratio index and the data redundancy and duplication rate.

4. The method for intelligent parsing and governance of unstructured data according to claim 3, characterized in that, The step of determining the value efficiency matching degree based on the data value activity coefficient and cost efficiency coefficient includes: Based on the data value activity coefficient and cost efficiency coefficient, the value efficiency matching degree is output through the matching model; The matching model is configured to process the data value activity coefficient and the cost efficiency coefficient through a balance function to output the value efficiency matching degree; the balance function is constructed to respond simultaneously to the synergy level of the activity coefficient and the efficiency coefficient and the degree of difference between them.

5. The method for intelligent parsing and governance of unstructured data according to claim 1, characterized in that, The steps for obtaining the data retention period compliance rate and lifecycle strategy coverage rate of unstructured data, and determining the compliance risk control coefficient, include: The data retention period compliance rate is obtained by comparing the amount of data that meets the retention period requirements with the total amount of data. The lifecycle strategy coverage is obtained by comparing the number of data types for which strategies have been defined with the total number of data types for which strategies should be defined. Based on the data retention period compliance rate and the lifecycle strategy coverage rate, the compliance risk control coefficient is calculated through risk control aggregation rules; the risk control aggregation rules are used to integrate the evaluation results of the compliance rate and the coverage rate, and the compliance risk control coefficient is positively correlated with both the data retention period compliance rate and the lifecycle strategy coverage rate.

6. The method for intelligent parsing and governance of unstructured data according to claim 5, characterized in that, The steps of determining the data lifecycle health based on value efficiency matching degree and compliance risk control coefficient, and outputting governance opinions corresponding to the data lifecycle health include: Based on preset association rules, the value efficiency matching degree is combined with the compliance risk control coefficient to output the data lifecycle health degree; The association rule defines the joint influence mechanism of the matching degree and the risk control coefficient on the data health status. The data lifecycle health is positively correlated with the value efficiency matching degree and the compliance risk control coefficient.

7. A system for intelligent parsing and governance of unstructured data, characterized in that, The system includes: The log data analysis module is used to establish a connection channel with the log database based on preset permissions, and to determine the data activity and data lineage integrity of unstructured data based on the log database. The data cost analysis module is used to obtain the storage cost-effectiveness ratio and data redundancy and duplication rate of unstructured data; The value-cost matching module is used to determine the data value activity coefficient based on data activity and data lineage integrity, determine the cost efficiency coefficient based on storage cost-benefit ratio and data redundancy and duplication rate, and determine the value-efficiency matching degree based on the data value activity coefficient and cost efficiency coefficient. The risk control coefficient determination module is used to obtain the data retention period compliance rate and lifecycle strategy coverage rate of unstructured data, and to determine the compliance risk control coefficient. The data health assessment module is used to assess the health of the data lifecycle based on the value-efficiency matching degree and compliance risk control coefficient, and output governance opinions corresponding to the data lifecycle health.

8. The intelligent analysis and governance system for unstructured data according to claim 7, characterized in that, The value-cost matching module includes: The activity analysis unit is used to perform max-min normalization on the current data activity level to obtain the data activity index. The data lineage analysis unit is used to process the ratio of the number of currently tracked lineage nodes to the total number of lineage nodes that should be tracked, and obtain the data lineage integrity index. The activity analysis unit is used to integrate the data activity index and the data lineage integrity index according to the value fusion rule to generate the data value activity coefficient; the value fusion rule defines the mapping relationship from data usage frequency and data lineage quality to data value activity, and the data value activity coefficient is positively correlated with both the data activity index and the data lineage integrity index.

9. The intelligent analysis and governance system for unstructured data according to claim 8, characterized in that, The value-cost matching module also includes: The cost-benefit ratio acquisition unit is used to process the ratio of total storage cost to data access revenue value to obtain the storage cost-benefit ratio; The normalization processing unit is used to perform maximum-min normalization on the current storage cost-benefit ratio to obtain the storage cost-benefit ratio index. The cost efficiency analysis unit is used to determine the cost efficiency coefficient based on the storage cost-effectiveness ratio index and the data redundancy and duplication rate according to the economic mapping rule; wherein the economic mapping rule is constructed such that the cost efficiency coefficient is negatively correlated with both the storage cost-effectiveness ratio index and the data redundancy and duplication rate.

10. The intelligent analysis and governance system for unstructured data according to claim 9, characterized in that, The value-cost matching module also includes: The matching execution unit is used to output the value-efficiency matching degree based on the data value activity coefficient and cost efficiency coefficient through the matching model. The matching model is configured to process the data value activity coefficient and the cost efficiency coefficient through a balance function to output the value efficiency matching degree; the balance function is constructed to respond simultaneously to the synergy level of the activity coefficient and the efficiency coefficient and the degree of difference between them.

Images